Consider using Cloudera Director for cluster provisioning

[laserson]

The Spark EC2 scripts have variable quality depending on instance types and options that are set. Also, they mainly just set up Spark, whereas we may want some of the other tools in the Hadoop stack (e.g., @tomwhite's partitioning tool that uses Crunch/Hadoop 2.x). Cloudera Director may make it more reliable to set up the whole stack, and also may more easily support using alternate clouds as well.

[tomwhite]

I created #45 for this. This change allows you to bring up a cluster using Cloudera Director - there's still more follow-on work to install eggo on the gateway node (need to change setup_master it to run as a non-root user).

[tomwhite]

Merged #45