Cleanup PIP package tools and installation documentation.

There are a few users to consider: 1. A developer on TFF wants source at HEAD. 2. A researcher using TFF wants pip package at release in 90% of scenarios and sometimes (ideally rarely) wants nightly, but never wants to build a pip package. 3. A developer making a framework using TFF wants pip release and nightly, but never wants to build a pip package. 4. A researcher upstreaming change to TFF basically has moved from google-parfait#2 to google-parfait#3. Fixes: #878 PiperOrigin-RevId: 321226038
amitport · Sep 6, 2020 · 2384cf2 · 2384cf2
1 parent 20dd647
commit 2384cf2
Show file tree

Hide file tree

Showing 7 changed files with 172 additions and 118 deletions.
diff --git a/docs/install.md b/docs/install.md
@@ -13,7 +13,7 @@ There are a few ways to set up your environment to use TensorFlow Federated
     [build TensorFlow Federated](#build-the-tensorflow-federated-pip-package)
     from source.
 
-## Install TensorFlow Federated using pip
+## Install TensorFlow Federated using `pip`
 
 #### 1. Install the Python development environment.
 
@@ -22,7 +22,7 @@ On Ubuntu:
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">sudo apt update</code>
 <code class="devsite-terminal">sudo apt install python3-dev python3-pip  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
 On macOS:
@@ -32,7 +32,7 @@ On macOS:
 <code class="devsite-terminal">export PATH="/usr/local/bin:/usr/local/sbin:$PATH"</code>
 <code class="devsite-terminal">brew update</code>
 <code class="devsite-terminal">brew install python  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
 #### 2. Create a virtual environment.
@@ -45,30 +45,39 @@ On macOS:
 
 Note: To exit the virtual environment, run `deactivate`.
 
-#### 3. Install the TensorFlow Federated `pip` package.
+#### 3. Install the TensorFlow Federated Python package.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal tfo-terminal-venv">pip install --upgrade tensorflow_federated</code>
 </pre>
 
-#### 4. (Optional) Test Tensorflow Federated.
+#### 4. Test Tensorflow Federated.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal tfo-terminal-venv">python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"</code>
 </pre>
 
-Success: TensorFlow Federated is now installed.
+Success: The latest TensorFlow Federated Python package is now installed.
 
-## Build the TensorFlow Federated pip package
+## Build the TensorFlow Federated Python package from source
 
-### 1. Install the Python development environment.
+Building a TensorFlow Federated Python package from source is helpful when you
+want to:
+
+*   Make changes to TensorFlow Federated and test those changes in a component
+    that uses TensorFlow Federated before those changes are submitted or
+    released.
+*   Use changes that have been submitted to TensorFlow Federated but have not
+    been released.
+
+#### 1. Install the Python development environment.
 
 On Ubuntu:
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">sudo apt update</code>
 <code class="devsite-terminal">sudo apt install python3-dev python3-pip  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
 On macOS:
@@ -78,118 +87,58 @@ On macOS:
 <code class="devsite-terminal">export PATH="/usr/local/bin:/usr/local/sbin:$PATH"</code>
 <code class="devsite-terminal">brew update</code>
 <code class="devsite-terminal">brew install python  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
-### 2. Install Bazel.
+#### 2. Install Bazel.
 
 [Install Bazel](https://docs.bazel.build/versions/master/install.html), the
 build tool used to compile Tensorflow Federated.
 
-### 3. Clone the Tensorflow Federated repository.
+#### 3. Clone the Tensorflow Federated repository.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">git clone https://github.com/tensorflow/federated.git</code>
 <code class="devsite-terminal">cd "federated"</code>
 </pre>
 
-### 4. Create a virtual environment.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">virtualenv --python python3 "venv"</code>
-<code class="devsite-terminal">source "venv/bin/activate"</code>
-<code class="devsite-terminal tfo-terminal-venv">pip install --upgrade pip</code>
-</pre>
-
-Note: To exit the virtual environment, run `deactivate`.
-
-### 5. Install Tensorflow Federated dependencies.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal tfo-terminal-venv">pip install --requirement "requirements.txt"</code>
-</pre>
-
-### 6. (Optional) Test Tensorflow Federated.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal tfo-terminal-venv">bazel test //tensorflow_federated/...</code>
-</pre>
-
-#### 7. Build the pip package.
+#### 4. Build the TensorFlow Federated Python package.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">mkdir "/tmp/tensorflow_federated"</code>
 <code class="devsite-terminal">bazel run //tensorflow_federated/tools/development:build_pip_package -- \
-    "/tmp/tensorflow_federated"</code>
+    --nightly \
+    --output_dir "/tmp/tensorflow_federated"</code>
 </pre>
 
-#### 8. Create a new project.
+#### 5. Create a new project.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">mkdir "/tmp/project"</code>
 <code class="devsite-terminal">cd "/tmp/project"</code>
+</pre>
 
+#### 6. Create a virtual environment.
+
+<pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">virtualenv --python python3 "venv"</code>
 <code class="devsite-terminal">source "venv/bin/activate"</code>
 <code class="devsite-terminal tfo-terminal-venv">pip install --upgrade pip</code>
 </pre>
 
 Note: To exit the virtual environment run `deactivate`.
 
-#### 9. Install the pip package.
+#### 7. Install the TensorFlow Federated Python package.
 
 <pre class="prettyprint lang-bsh">
-<code class="devsite-terminal tfo-terminal-venv">pip install --upgrade "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"</code>
+<code class="devsite-terminal tfo-terminal-venv">pip install --upgrade "/tmp/tensorflow_federated/"*".whl"</code>
 </pre>
 
-#### 10. Test Tensorflow Federated.
+#### 8. Test Tensorflow Federated.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal tfo-terminal-venv">python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"</code>
 </pre>
 
-Success: The TensorFlow Federated package is built.
-
-## Using Docker
-
-Create a Tensorflow Federated development environment using Docker on Ubuntu or
-macOS.
-
-### 1. Install Docker.
-
-[Install Docker](https://docs.docker.com/install/) on your local machine.
-
-### 2. Clone the latest Tensorflow Federated source.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">git clone https://github.com/tensorflow/federated.git</code>
-<code class="devsite-terminal">cd "federated"</code>
-</pre>
-
-### 3. Build a Docker image.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">docker build . \
-    --tag tensorflow_federated</code>
-</pre>
-
-### 4. Start a Docker container.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">docker run \
-    --interactive \
-    --tty \
-    --volume $(pwd):/federated \
-    --workdir /federated \
-    tensorflow_federated \
-    bash</code>
-</pre>
-
-### 5. (Optional) Test Tensorflow Federated.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">bazel test //tensorflow_federated/...</code>
-</pre>
-
-Success: The TensorFlow Federated development environment is ready, now
-[build the pip package](#build-the-pip-package).
+Success: A TensorFlow Federated Python package is now built from source and
+installed.
diff --git a/tensorflow_federated/tools/client/build_image.sh b/tensorflow_federated/tools/client/build_image.sh
@@ -25,19 +25,21 @@
 set -e
 
 main() {
+  # Parse arguments
   local artifacts_dir="$1"
 
   if [[ -z "${artifacts_dir}" ]]; then
     artifacts_dir="$(mktemp -d)"
     trap "rm -rf ${artifacts_dir}" EXIT
   fi
 
+  # Create working directory
   cp -LR "tensorflow_federated" "${artifacts_dir}"
   pushd "${artifacts_dir}"
 
   # Build the TensorFlow Federated package
   tensorflow_federated/tools/development/build_pip_package \
-      "${artifacts_dir}"
+      --output_dir "${artifacts_dir}"
 
   # Build the TensorFlow Federated runtime image
   docker build \

diff --git a/tensorflow_federated/tools/development/build_pip_package.sh b/tensorflow_federated/tools/development/build_pip_package.sh
@@ -17,30 +17,62 @@
 #
 # Usage:
 #   bazel run //tensorflow_federated/tools/development:build_pip_package -- \
-#       "/tmp/tensorflow_federated"
+#       --output_dir "/tmp/tensorflow_federated"
+#   bazel run //tensorflow_federated/tools/development:build_pip_package -- \
+#       --nightly \
+#       --output_dir "/tmp/tensorflow_federated"
 #
 # Arguments:
+#   nightly: A flag indicating whether or not to build the nightly version of
+#     the pip package.
 #   output_dir: An output directory.
-#   project_name: A project name, defaults to `tensorflow_federated`.
 set -e
 
 die() {
   echo >&2 "$@"
   exit 1
 }
 
+usage() {
+  local script_name=$(basename "${0}")
+  echo "usage: ${script_name} [--nightly] [--output_dir PATH]" 1>&2
+}
+
 main() {
-  local output_dir="$1"
-  local project_name="$2"
+  # Parse arguments
+  local nightly=0
+  local output_dir=""
 
-  if [[ ! -d "${output_dir}" ]]; then
-    die "The output directory '${output_dir}' does not exist."
+  while [[ "$#" -gt 0 ]]; do
+    opt="$1"
+    case "${opt}" in
+      --nightly)
+        nightly="1"
+        shift
+        ;;
+      --output_dir)
+        output_dir="$2"
+        shift
+        # Shift might exit with an error code if no output_dir was provided.
+        shift || break
+        ;;
+      *)
+        usage
+        exit 1
+        ;;
+    esac
+  done
+
+  if [[ -z ${output_dir} ]]; then
+    usage
+    exit 1
   fi
 
-  if [[ -z "${project_name}" ]]; then
-    project_name="tensorflow_federated"
+  if [[ ! -d "${output_dir}" ]]; then
+    die "The output directory '${output_dir}' does not exist."
   fi
 
+  # Create working directory
   local temp_dir="$(mktemp -d)"
   trap "rm -rf ${temp_dir}" EXIT
   cp -LR "tensorflow_federated" "${temp_dir}"
@@ -52,10 +84,14 @@ main() {
   pip install --upgrade pip
 
   # Build pip package
+  flags=()
+  if [[ ${nightly} == "1" ]]; then
+    flags+=("--nightly")
+  fi
   pip install --upgrade setuptools wheel
   python "tensorflow_federated/tools/development/setup.py" bdist_wheel \
       --universal \
-      --project_name "${project_name}"
+      "${flags[@]}"
   popd
 
   cp "${temp_dir}/dist/"* "${output_dir}"

diff --git a/tensorflow_federated/tools/development/publish_pip_package.sh b/tensorflow_federated/tools/development/publish_pip_package.sh
@@ -17,7 +17,7 @@
 #
 # Usage:
 #   bazel run //tensorflow_federated/tools/development:publish_pip_package -- \
-#       "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"
+#       --package "/tmp/tensorflow_federated/"*".whl"
 #
 # Arguments:
 #   package: A path to a local pip package.
@@ -28,13 +28,41 @@ die() {
   exit 1
 }
 
+usage() {
+  local script_name=$(basename "${0}")
+  echo "usage: ${script_name} [--package PATH]" 1>&2
+}
+
 main() {
-  local package="$1"
+  # Parse arguments
+  local package=""
+
+  while [[ "$#" -gt 0 ]]; do
+    opt="$1"
+    case "${opt}" in
+      --package)
+        package="$2"
+        shift
+        # Shift might exit with an error code if no output_dir was provided.
+        shift || break
+        ;;
+      *)
+        usage
+        exit 1
+        ;;
+    esac
+  done
+
+  if [[ -z ${package} ]]; then
+    usage
+    exit 1
+  fi
 
   if [[ ! -f "${package}" ]]; then
     die "The package '${package}' does not exist."
   fi
 
+  # Create working directory
   local temp_dir="$(mktemp -d)"
   trap "rm -rf ${temp_dir}" EXIT
   pushd "${temp_dir}"