Release Notes

We’ve been hard at work in Wallaroo Labs-ville with a whole slew of features & bug fixes just to help you succeed with Wallaroo! We’ve added support for dynamic keys -- now you aren’t required to pre-define your partition keys when you’re writing your app! Let us do the heavy lifting for you!

This release has breaking changes and some new additions! We’ve added a table of contents below to help you navigate to sections you might find relevant.

Breaking Changes
- Python API
- Go API
New Additions
How to Upgrade Your Application
Upgrading Wallaroo
CHANGELOG

Breaking Changes

Unfortunately, we had to break some things as part of adding dynamic keys. Below is a list of changes you might need to make as part of upgrading.

Python API

The to_state_partition_u64 method has been removed and should be replaced with to_state_partition. If you were using the to_state_partition_u64 method your partition_keys will also have to be updated from a list of non-negative integers to a list of strings.

Example

Our Python Market Spread example application, previously used to_state_partition_u64 in the ApplicationBuilder:

.to_state_partition_u64(
            update_market_data, SymbolData, "symbol-data",
            symbol_partition_function, symbol_partitions
        )

This method has been removed and is replaced with to_state_partition:

.to_state_partition(
            update_market_data, SymbolData, "symbol-data",
            symbol_partition_function, symbol_partitions
        )

Since the partition_keys were previously defined
using the str_to_partition method to convert the string data to non-negative integers, that too needed to be updated from this:

symbol_partitions = [str_to_partition(x.rjust(4)) for x in
                         load_valid_symbols()]

To this, which is a list of strings:

symbol_partitions = [x.rjust(4) for x in
                         load_valid_symbols()]

See the full application code here.

Go API

The ToStatePartition and ToStatePartitionMulti methods have been updated to no longer accept partition keys as a parameter. If you wish to continue using your partition keys, you’ll need to replace them with ToStatePartitionWithKeys and ToStatePartitionMultiWithKeys methods respectively.

The type of your partition keys will also need updating, you will need to change your partition methods to return a byte slice instead of a U64 slice.

Example

Our Go Alphabet example has been updated to replace ToStatePartition:

ToStatePartition(&AddVotes{}, &RunningVotesTotalBuilder{}, "running vote totals", &LetterPartitionFunction{}, MakeLetterPartitions())

with ToStatePartitionWithKeys:

ToStatePartitionWithKeys(&AddVotes{}, &RunningVotesTotalBuilder{}, "running vote totals", &LetterPartitionFunction{}, MakeLetterPartitions())

And the partition method was updated from returning a uint64 slice:

func (lpf *LetterPartitionFunction) Partition(data interface{}) []uint64 {
	lav := data.(*LetterAndVotes)
	return []uint64{lav.Letter}
}

To a byte slice:

func (lpf *LetterPartitionFunction) Partition(data interface{}) []byte {
	lav := data.(*LetterAndVotes)
	return []byte{lav.Letter}
}

See the full application code here.

New Additions

Here are some new features we've added to Wallaroo!

Dynamic Keys

You’re now able to let Wallaroo generate your partition keys as needed, rather than needing to declare all of them up front. We've covered in detail how to update your application to do in the Updating your application section.

Cluster Shrinker Tool

We’ve added a new tool to help operators shrink a Wallaroo cluster. It’s appropriately called the cluster_shrinker and instructions on its usage can be found in the Shrink to Fit section of our Autoscale documentation.

Support for Multiple Sinks per Pipeline

Pipelines now support multiple sinks! Check out the updated Python API docs and Go API docs for information on how to add multiple sinks to your Wallaroo application.

How to Upgrade your Application

So, will you be transitioning your application to use dynamic keys? Or will you continue to declare your partition keys in advance?

I like new things! Or, I need dynamic keys

Are you using the Go API, or the Python API?

Go

You’ll need to update some methods:
The ToStatePartition method no longer has a parameter for partition keys, so you’ll want to continue using this function. You’ll need to update all instances of it, though, to no longer pass in your partition keys. Similarly, ToStatePartitionMulti no longer has a parameter for partition keys, and you must ensure that you no longer pass any partition keys to it.

So for our Word Count example, the ToStatePartition method will change from this, without dynamic keys:

ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition())

To this, with dynamic keys:

ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{})

Python

In Python, the to_state_partition method has been updated to use an optional parameter for your partition keys. So, you don’t need to modify the method itself. However, since you want to use dynamic keys now, you’ll need to no longer send in your partition keys. To do this, you’ll have to update the methods’ arguments so that you no longer pass in your keys.

So for our Word Count examples, the to_state_partition method will change from this, without dynamic keys:

ab.to_state_partition(count_word, WordTotals, "word totals",
        partition, word_partitions)

To this, with dynamic keys:

ab.to_state_partition(count_word, WordTotal, "word totals",
        partition)

I know my keys in advance and want a performance bump

Are you using the Go API, or the Python API?

Go

You’ll need to update some methods:
ToStatePartition should be ToStatePartitionWithKeys and ToStatePartitionMulti should become ToStatePartitionMultiWithKeys.

So for our Word Count example, the ToStatePartition method will change from this, without dynamic keys:

ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition())

To this, with dynamic keys:

ToStatePartitionWithKeys(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition())

Python

In Python, the to_state_partition method has been updated to use an optional parameter for your partition keys. So, you don’t need to modify the method itself. However, the keys now must be strings. Previously, keys could be any object, as long as it could be hashed, and checked for equality.

So if you previously partitioned on say, a list of integers:

partitions = [1,2,3,4,5,6,7,8,9,10]

You’ll want to update the partitions to a list of strings instead, like so:

partitions = ["1","2","3","4","5","6","7","8","9","10"]

And with those changes, you should be ready to get started with Wallaroo 0.5.0!

Upgrading Wallaroo

In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, or our mailing list.
We love questions!

If you have made no changes to Wallaroo or Pony since installation, your best bet will be to delete your Wallaroo installation and start from scratch, following the [instructions] (https://docs.wallaroolabs.com/book/getting-started/choosing-an-installation-option.html) of your choice.

Below are instructions for Upgrading Wallaroo when compiled from source, Upgrading Wallaroo in Docker, and Upgrading Wallaroo in Vagrant.

Upgrading Wallaroo when compiled from source

These instructions are for Ubuntu Linux. It's assumed that if you are using a different operating system then you are able to translate these instructions to your OS of choice.

Upgrading ponyc to 0.24.0

ponyc can be upgraded with the following command:

sudo apt-get install --only-upgrade ponyc=0.24.0

Verify you are now on the correct version of ponyc by running:

ponyc --version

You should get the following output:

0.24.0 [release]

Upgrading Pony Stable

stable can be upgraded with the following command:

sudo apt-get install --only-upgrade pony-stable=0.1.4-121.d8a403e

Verify you are now on the correct version of pony-stable by running:

stable version

You should get the following output:

0.1.4-d8a403e [release]

How to Upgrade Wallaroo

Once you're on the latest ponyc and pony stable, you're ready to switch over to Wallaroo 0.5.0.

We recommend moving your current Wallaroo directory and starting with a fresh clone of the latest release. If you have made prior changes to the Wallaroo code, you’ll need to re-implement those changes. To get a fresh clone, assuming that you cloned the repository to the directory we recommended in setup, you’ll need to run the following:

cd ~/wallaroo-tutorial/
mv wallaroo/ wallaroo-0.4.x/

To get a new copy of the Wallaroo repository, run the following commands:

cd ~/wallaroo-tutorial/
git clone https://github.com/wallaroolabs/wallaroo
cd wallaroo
git checkout 0.5.0

You can then run the following commands to build the necessary tools to continue developing using Wallaroo 0.5.0:

cd ~/wallaroo-tutorial/wallaroo
make build-machida build-giles-all build-utils-cluster_shutdown

Upgrading the Wallaroo Docker image

To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.

docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.5.0

Upgrading Wallaroo Source Code

If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

UNIX & MacOS Users

For UNIX users, you can move the directory with the following command:

mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.4.x-src/

Windows Users

For Windows users, you can move the directory with the following command:

move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.4.x-src

Once done moving, you can re-create the wallaroo-src directory with the following command:

mkdir c:\wallaroo-docker\wallaroo-src

Upgrading Wallaroo in Vagrant

To upgrade your Wallaroo installation in Vagrant, you’ll want to start by moving your current Vagrant directory to a new location. Assuming that you’ve installed it according to our setup documentation, you’ll run:

cd ~/
mv wallaroo-tutorial/wallaroo/vagrant wallaroo-tutorial/wallaroo/vagrant-0.4.x
cd ~/wallaroo-tutorial/wallaroo/
git fetch origin
git checkout -f 0.5.0

Finally, to provision your new Vagrant box, run the following commands:

cd ~/wallaroo-tutorial/wallaroo/vagrant
vagrant up

If you have modified your old Vagrant VM in any way that you intend to persist, you’ll need to do that now. For example, copy any edited or new files from the old Vagrant VM to the new one. When you’ve completed that, it’s a good idea to clean up your old Vagrant box, by running:

cd ~/wallaroo-tutorial/wallaroo/vagrant-0.4.x
vagrant destroy

CHANGELOG

[0.5.0] - 2018-07-25

Fixed

Filter None values in machida compute_multi results (PR #2179)
Fix busy-loop/scheduler yield bugs in TCPConnection-related classes (PR #2142)
Fix errors in Python decorator API documentation (PR #2124)
Improve autoscale performance and reliability, and fix related edge case bugs (PR #2122)
Fix bug that caused Wallaroo to shut down when connecting a new source after a 3->2 shrink (PR #2072)
Correctly remove boundary references when worker leaves (PR #2073)
Correctly update routers after shrink (PR #2018)
Only try to shrink if supplied worker names are still in the cluster (PR #2034)
Fix bug when using cluster_shrinker with non-initializer (PR #2011)
Ensure that all running workers migrate to joiners (PR #2027)
Clean up recovery files during shrink (PR #2012)
Ensure that new sources don't try to connect to old workers (PR #2004)
Fail when control channel can't listen (PR #1982)
Only create output file for Giles sender when writing (PR #1964)

Added

Add support for dynamic keys (PR #2265)
Inform joining worker to shut down on join error (PR #2086)
Add partition count observability query (PR #2081)
Add support for multiple sinks per pipeline (PR #2060)
Allow joined worker to recover with original command line (PR #1933)
Add support for query requesting information about partition step distribution across workers (PR #2025)
Add tool to allow an operator to shrink a Wallaroo cluster (PR #2005)

Changed

Clean up external message protocol (PR #2032)
Remove "name()" from StateBuilder interface (PR #1988)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.5.0

Release Notes

Table of Contents

Breaking Changes

Python API

Example

Go API

Example

New Additions

Dynamic Keys

Cluster Shrinker Tool

Support for Multiple Sinks per Pipeline

How to Upgrade your Application

I like new things! Or, I need dynamic keys

Go

Python

I know my keys in advance and want a performance bump

Go

Python

Upgrading Wallaroo

Upgrading Wallaroo when compiled from source

Upgrading ponyc to 0.24.0

Upgrading Pony Stable

How to Upgrade Wallaroo

Upgrading the Wallaroo Docker image

Upgrading Wallaroo Source Code

UNIX & MacOS Users

Windows Users

Upgrading Wallaroo in Vagrant

CHANGELOG

[0.5.0] - 2018-07-25

Fixed

Added

Changed