Skip to content

Commit

Permalink
Change Python API to use decorators
Browse files Browse the repository at this point in the history
The Python API was using a ``class-based'' approach that was very
similar to the C++ API. This wasn't very pythonic, so this change adds
decorators in place of a lot of the boiler-plate classes we were using.
This results in shorter application code, and less responsibility on the
developer to respect Wallaroo interfaces.
  • Loading branch information
Alan Mosca committed Jan 2, 2018
1 parent e8aa9fc commit fa5e8ca
Show file tree
Hide file tree
Showing 16 changed files with 519 additions and 636 deletions.
69 changes: 23 additions & 46 deletions book/core-concepts/decoders-and-encoders.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,74 +59,51 @@ Let's make that a bit more concrete. Let's say we are sending in two messages; e
Wallaroo's `TCPSource` takes a `Decoder` that can process a framed message protocol. You'll need to implement three methods. Below is a `Decoder` that we can use to process our stream of strings that we layed out in the previous section.

```python
class Decoder(object):
def header_length(self):
return 4

def payload_length(self, bs):
return struct.unpack(">I", bs)[0]

def decode(self, bs):
return bs.decode("utf-8")
@wallaroo.decoder(header_length=4, length_fmt=">I")
def decode(self, bs):
return bs.decode("utf-8")
```

`header_length` will return the fixed size of our payload field.
`header_length` is the fixed size of our payload field.

`payload_length` is used to decode our message header to determine how long the payload is going to be. In this case, we are relying on the Python `struct` package to decode the bytes. If you aren't familiar with `struct`, you can check out [the documentation](https://docs.python.org/2/library/struct.html) to learn more. Remember, when using `struct`, don't forget to import it!
`length_fmt` is used internally to decode our message header to determine how long the payload is going to be. We rely on the Python `struct` package to decode the bytes. If you aren't familiar with `struct`, you can check out [the documentation](https://docs.python.org/2/library/struct.html) to learn more. Remember, when using `struct`, don't forget to import it!

`decode` takes a series of bytes that represent your payload and turns it into an application message. In this case, our application message is a string, so we take the incoming byte stream `bs` and convert it to UTF-8 Python string.

Here's a slightly more complicated example taken from our [Alphabet Popularity Contest example](https://github.com/WallarooLabs/wallaroo/tree/{{ book.wallaroo_version }}/examples/python/alphabet).

```python
class Decoder(object):
def header_length(self):
return 4

def payload_length(self, bs):
return struct.unpack(">I", bs)[0]

def decode(self, bs):
(letter, vote_count) = struct.unpack(">sI", bs)
return Votes(letter, vote_count)
```

Like we did in our previous example, we're using a 4-byte payload length header:

```python
class Decoder(object):
def header_length(self):
return 4
@wallaroo.decoder(header_length=4, length_fmt=">I")
def decode(self, bs):
(letter, vote_count) = struct.unpack(">sI", bs)
return Votes(letter, vote_count)
```

We're once again use `struct` to decode our header into a payload length:
Like we did in our previous example, we're using a 4-byte integer payload length header:

```python
class Decoder(object):
def payload_length(self, bs):
return struct.unpack(">I", bs)[0]
@wallaroo.decoder(header_length=4, length_fmt=">I")
```

But this time, we are doing something slightly more complicated in our payload. Our payload is two items, a string representing a letter and some votes for that letter. We'll unpack those using `struct` and create a domain specific object `Votes` to return.

```python
class Decoder(object):
def decode(self, bs):
(letter, vote_count) = struct.unpack(">sI", bs)
return Votes(letter, vote_count)
def decode(self, bs):
(letter, vote_count) = struct.unpack(">sI", bs)
return Votes(letter, vote_count)
```

## Creating an Encoder

Wallaroo encoders do the opposite of decoders. A decoder takes a stream of bytes and turns in into messages for Wallaroo to process. An encoder receives a stream of messages and converts them into a stream of bytes for sending to another system.

Here's a quick example `Encoder`:
Here's a quick example encoder:

```python
class Encoder(object):
def encode(self, data):
# data is a string
return data + "\n"
@wallaroo.encoder
def encode(self, data):
# data is a string
return data + "\n"
```

This is just about the simplest encoder you could have. It's from the [Reverse Word example](https://github.com/WallarooLabs/wallaroo/tree/{{ book.wallaroo_version }}/examples/python/reverse). It takes a string that we want to send to an external system as an input, adds a newline at the end and returns it for sending.
Expand All @@ -139,10 +116,10 @@ class Votes(object):
self.letter = letter
self.votes = votes

class Encoder(object):
def encode(self, data):
# data is a Votes
return struct.pack(">IsQ", 9, data.letter, data.votes)
@wallaroo.encoder
def encode(self, data):
# data is a Votes
return struct.pack(">IsQ", 9, data.letter, data.votes)
```

Let's take a look at what is happening here. First of all, we are once again using the Python `struct` package. In this case, though, we are creating our own packed binary message. It's a framed message with a 4-byte header for the payload length, plus the payload:
Expand Down
18 changes: 10 additions & 8 deletions book/core-concepts/partitioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ class Stock(object):
self.price = price


@wallaroo.state
class Stocks(object):
def __init__(self):
self.stocks = {}
Expand Down Expand Up @@ -46,6 +47,7 @@ To do this, a _partition function_ is used to determine which _state part_ a par
In order to take advantage of state partitioning, state objects need to be broken down. In the stock example there is already a class that represents an individual stock, so it can be used as the partitioned state.

```python
@wallaroo.state
class Stock(object):
def __init__(self, symbol, price):
self.symbol = symbol
Expand All @@ -55,12 +57,12 @@ class Stock(object):
Since the computation only has one stock in its state now, there is no need to do a dictionary look up. Instead, the computation can update the particular Stock's state right away.

```python
class UpdateStock(object):
def compute(self, stock, state):
state.symbol = stock.symbol
state.price = stock.price
@wallaroo.state_computation
def compute(self, stock, state):
state.symbol = stock.symbol
state.price = stock.price

return (None, True)
return (None, True)
```

### Partition Key
Expand All @@ -73,7 +75,7 @@ Currently, the partition keys for a particular partition need to be defined alon
The partition function takes in message data and returns a partition key. In the example, the message symbol would be extracted from the message data and returned as the key.

```python
class SymbolPartitionFunction(object):
def partition(self, data):
return data.symbol
@wallaroo.partition
def partition(self, data):
return data.symbol
```
1 change: 1 addition & 0 deletions book/core-concepts/state.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ So what is a state object? Let's explain via an example. Imagine we are writing
Wallaroo allows you to define your own state objects that match your domain. For example, our example application might define a state object as:

```python
@wallaroo.state
class Stock(object):
def __init__(self, symbol, price):
self.symbol = symbol
Expand Down
1 change: 1 addition & 0 deletions book/core-concepts/working-with-state.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Wallaroo's state objects allow developers to define their own domain-specific da
Imagine a word counting application. We'll have a state object for each different word. Each word and count object would look something like:

```python
@wallaroo.state
class WordAndCount(object):
def __init__(self, word="", count=0):
self.word = word
Expand Down
Loading

0 comments on commit fa5e8ca

Please sign in to comment.