SynthTool (for client libraries)

This tool helps to generate and layout cloud client libraries. Synthtool runs the GAPIC (Generated API Client) Generator via Google API Artifact Manager (artman).

Installation

This tool requires Python 3.6. Either install it from python.org or use pyenv to get 3.6.

If you are using GAPIC generator synthtool.gcp.GAPICGenerator or DiscoGAPIC generator synthtool.gcp.DiscoGAPICGenerator, install Docker.

Install latest

python3 -m pip install --user --upgrade git+https://github.com/googleapis/synthtool.git

Basic usage

To start the process of generation, clone the destination repository.

git clone [email protected]:googleapis/google-cloud-python.git
cd google-cloud-python/

Navigate to the destination directory to generate the library.

cd tasks/

Running synthtool

If a synth.py script is not present, create a new one.

You can create one from scratch or copy one from another library.

• e.g. the synth.py for the Cloud Tasks API for Python, Java, Node.js, PHP, or Ruby.

Run synthtool:

python3 -m synthtool

After synthtool runs successfully:

• Investigate the changes it made
• Run the library tests
• Commit and push the changes to a branch and open a Pull Request

Find examples below in different programming languages (Cloud Tasks API used as an example).

Python

• Clone the destination repository:

  git clone [email protected]:googleapis/google-cloud-python.git
  cd google-cloud-python/
  

• Navigate to the destination directory to generate the library:

  cd tasks/
  

• Run synthtool to generate using the existing synth.py file for the Python Client for Cloud Tasks API:

  python3 -m synthtool
  

• See the Python Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Java

• Clone the destination repository:

  git clone [email protected]:googleapis/google-cloud-java.git
  cd google-cloud-java/
  

• Navigate to the destination directory to generate the library:

  cd google-cloud-clients/google-cloud-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Google Cloud Java Client for Cloud Tasks:

  python3 -m synthtool
  

• See the Java Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Node.js

• Clone the destination repository:

  git clone [email protected]:googleapis/nodejs-tasks.git
  cd nodejs-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Google Cloud Tasks Node.js Client:

  python3 -m synthtool
  

• See the Node.js Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

PHP

• Clone the destination repository:

  git clone [email protected]:googleapis/google-cloud-php.git
  cd google-cloud-php/
  

• Navigate to the destination directory to generate the library:

  cd Tasks/
  

• Run synthtool to generate using the existing synth.py file for the Google Cloud Tasks client for PHP:

  python3 -m synthtool
  

• See the PHP Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Ruby

• Clone the destination repository:

  git clone [email protected]:googleapis/google-cloud-ruby.git
  cd google-cloud-ruby/
  

• Navigate to the destination directory to generate the library:

  cd google-cloud-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Ruby Client for Cloud Tasks API:

  python3 -m synthtool
  

• See the Ruby Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Features

Templating

SynthTool supports template files using Jinja.

Templates are found in subdirectories of synthtool/gcp/templates/ for each language,

• e.g. the template directories for Python, Node.js, PHP, or Ruby.

You can generate and copy templates using gcp.CommonTemplates in your synth.py:

common_templates = gcp.CommonTemplates()

templates = common_templates.node_library()
s.copy(templates)

You can provide variables to templates as keyword arguments to the library generation method:

common_templates = gcp.CommonTemplates()

templates = common_templates.node_library(version=5, show_version=True, previous_versions=[1,2,3,4])

s.copy(templates)

Template files can access any values provided, e.g.

• README.md.j2

  {% if show_version %}
  The version is {{ version }}
  
  {% if previous versions is defined %}
  Previous versions:
    {% for ver in previous_versions %}
    - {{ ver }}
    {% endfor %}
  {% endif %}
  {% endif %}

You can learn more about Jinga templating in the Template Designer Documentation.

googleapis-private

SynthTool supports generation from googleapis/googleapis-private.

gapic = gcp.GAPICGenerator()

library = gapic.node_library('speech', 'v1', private=True)

2FA is required to clone a private repo.

• Using SSH: Before running Synthtool, set the environment variable AUTOSYNTH_USE_SSH to true.

The repo will be cloned using SSH.

• Using HTTPS: Generate a GitHub Personal Access Token with scope repo. Run synthtool.

When GitHub prompts for your GitHub password, provide the access token instead.

synthtool > Cloning googleapis-private.
Username for 'https://github.com': busunkim96
Password for 'https://[email protected]':

Artman Version

SynthTool uses the latest version of the Artman Docker image. You can change this by setting the environment variable SYNTHTOOL_ARTMAN_VERSION to the desired version tag.

export SYNTHTOOL_ARTMAN_VERSION=0.16.2

Local Googleapis

SynthTool supports generation from a local copy of googleapis. Specify the path to googleapis in the environment variable SYNTHTOOL_GOOGLEAPIS.

export SYNTHTOOL_GOOGLEAPIS=path/to/local/googleapis

Local GAPIC Generator

SynthTool supports generation from a local copy of gapic-generator. Specify the path to gapic-generator in the environment variable SYNTHTOOL_GENERATOR.

export SYNTHTOOL_GENERATOR=path/to/local/gapic-generator

Don't forget to compile gapic-generator before running SynthTool.

cd path/to/local/gapic-generator
./gradlew fatJar

Local Template Files

SynthTool supports specifying a local directory of templates. Specify the patht to the root template directory (not a SynthTool clone) in the environment variable SYNTHTOOL_TEMPLATES.

export SYNTHTOOL_TEMPLATES=path/to/local/templates

Include .proto files

SynthTool supports copying .proto API definition files from googleapis.

gapic = gcp.GAPICGenerator()

library = gapic.node_library('speech', 'v1', include_protos=True)

Context-Aware Commits

Autosynth runs synthtool on your synth.py nightly or more frequently. By default, it runs synthtool once, and if the generated code differs, creates a PR with the differences.

Autosynth can also find which changes in upstream repositories triggered changes in the generated code. To enable this behavior (context-aware commits), set one or both of the following flags in you synth.py file:

AUTOSYNTH_MULTIPLE_COMMITS
AUTOSYNTH_MULTIPLE_PRS

Example

Assume that since the library source code was last generated, A, B and X, Y were committed to googleapis and synthtool respectively, and they all triggered changes in the generated library code.

googleapis	synthtool (templates)
A	X
B	Y

Here's what autosynth will generate for each flag setting.

AUTOSYNTH_MULTIPLE_COMMITS = True

Autosynth will create one PR, with a single commit for each original commit:

PR
A
B
X
Y

---

AUTOSYNTH_MULTIPLE_COMMITS = True
AUTOSYNTH_MULTIPLE_PRS = True

Autosynth will create two PRs, with a single commit for each original commit:

PR1
A
B

PR2
X
Y

---

AUTOSYNTH_MULTIPLE_PRS = True

Autosynth will create two PRs, with a single commit combining all the original commits.

PR1
AB

PR2
XY

Helpful tips

Where does the generated code go?

SynthTool will run Artman which will create generated code that can be found at ~/.cache/synthtool/googleapis<-private>/artman_genfiles. This is useful for figuring out what it is you need to copy for your specific library.