Skip to content

Commit

Permalink
Merge branch 'main' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
ReubenFrankel authored Jan 31, 2025
2 parents 7b67928 + a609f4b commit 02cabc0
Show file tree
Hide file tree
Showing 11 changed files with 372 additions and 103 deletions.
4 changes: 2 additions & 2 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ Type hints allow us to spend less time reading documentation. Public modules are

All public modules in the SDK are checked for the presence of docstrings in classes and functions. We follow the [Google Style convention](https://www.sphinx-doc.org/en/master/usage/extensions/example_google.html) for Python docstrings so functions are required to have a description of every argument and the return value, if applicable.

### What is Poetry and why do we need it?
### What is uv and why do we need it?

For more info on `Poetry` and `Pipx`, please see the topic in our
For more info on `uv`, please see the topic in our
[python tips](./python_tips.md) guide.
208 changes: 163 additions & 45 deletions docs/cli_commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,86 +7,219 @@ prefix `poetry run`.

- Note: CLI mapping is performed in `pyproject.toml` and shims are recreated during `poetry install`:

````{tab} Poetry
```toml
...
[tool.poetry.scripts]
tap-mysource = 'singer_sdk.tests.sample_tap_parquet.parquet_tap:cli'
```
````
````{tab} uv
```toml
...
[project.scripts]
tap-mysource = 'singer_sdk.tests.sample_tap_parquet.parquet_tap:cli'
```
````
The CLI commands defined here will be configured automatically when the python library is installed by a user.
## For example, to run `--help`
````{tab} Poetry
```bash
poetry install && \
poetry run tap-mysource --help
```
````

````{tab} uv
```bash
uv sync && \
uv run tap-mysource --help
```
````

## Run in sync mode with auto-discovery

````{tab} Poetry
```bash
poetry install && \
poetry run tap-mysource \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
```
````

````{tab} uv
```bash
uv sync && \
uv run tap-mysource \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
````

## Run in sync mode with a catalog file input

````{tab} Poetry
```bash
poetry install && \
poetry run tap-mysource \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
--catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json
```
````

````{tab} uv
```bash
uv sync && \
uv run tap-mysource \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
--catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json
````

## Run in discovery mode

````{tab} Poetry
```bash
poetry install && \
poetry run tap-mysource --discover \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
```
````

````{tab} uv
```bash
uv sync && \
uv run tap-mysource --discover \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
````

## Run in discovery mode with a passed catalog file

````{tab} Poetry
```bash
poetry install && \
poetry run tap-mysource --discover \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json \
--catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json
```
````

````{tab} uv
```bash
uv sync && \
uv run tap-mysource --discover \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json \
--catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json
````

## Test connectivity

The `--test` option allows the user to validate configuration and assess connectivity.

````{tab} Poetry
```bash
poetry install && \
poetry run tap-mysource --test \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
```
````

````{tab} uv
```bash
uv sync && \
uv run tap-mysource --test \
--config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json
````

## Package Information

The `--about` option displays metadata about the package.

````{tab} Poetry
```console
$ poetry run sdk-tap-countries-sample --about
Name: sample-tap-countries
Description: Sample tap for Countries GraphQL API.
Version: [could not be detected]
SDK Version: 0.44.2
Capabilities:
- catalog
- state
- discover
- about
- stream-maps
- schema-flattening
- batch
Settings:
- Name: stream_maps
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_STREAM_MAPS
- Name: stream_map_config
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_STREAM_MAP_CONFIG
- Name: faker_config
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_FAKER_CONFIG
- Name: flattening_enabled
Type: ['boolean', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_FLATTENING_ENABLED
- Name: flattening_max_depth
Type: ['integer', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_FLATTENING_MAX_DEPTH
- Name: batch_config
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_BATCH_CONFIG
```
````

````{tab} uv
```console
$ uv run sdk-tap-countries-sample --about
Name: sample-tap-countries
Description: Sample tap for Countries GraphQL API.
Version: [could not be detected]
Sdk_Version: 0.3.5
Capabilities: ['sync', 'catalog', 'state', 'discover']
Settings: {'type': 'object', 'properties': {}}
SDK Version: 0.44.2
Capabilities:
- catalog
- state
- discover
- about
- stream-maps
- schema-flattening
- batch
Settings:
- Name: stream_maps
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_STREAM_MAPS
- Name: stream_map_config
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_STREAM_MAP_CONFIG
- Name: faker_config
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_FAKER_CONFIG
- Name: flattening_enabled
Type: ['boolean', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_FLATTENING_ENABLED
- Name: flattening_max_depth
Type: ['integer', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_FLATTENING_MAX_DEPTH
- Name: batch_config
Type: ['object', 'null']
Environment Variable: SAMPLE_TAP_COUNTRIES_BATCH_CONFIG
```
````

This information can also be printed in JSON format for consumption by other applications

````{tab} Poetry
```console
$ poetry run sdk-tap-countries-sample --about --format json
{
"name": "sample-tap-countries",
"description": "Sample tap for Countries GraphQL API.",
"version": "[could not be detected]",
"sdk_version": "0.3.5",
"sdk_version": "0.44.2",
"capabilities": [
"sync",
"catalog",
Expand All @@ -99,6 +232,29 @@ $ poetry run sdk-tap-countries-sample --about --format json
}
}
```
````

````{tab} uv
```console
$ uv run sdk-tap-countries-sample --about --format json
{
"name": "sample-tap-countries",
"description": "Sample tap for Countries GraphQL API.",
"version": "[could not be detected]",
"sdk_version": "0.44.2",
"capabilities": [
"sync",
"catalog",
"state",
"discover"
],
"settings": {
"type": "object",
"properties": {}
}
}
```
````

## Invocation options

Expand All @@ -108,47 +264,9 @@ There a are few options available to invoke your connector.

Activate the Poetry environment with `poetry shell` or prefix all your commands with `poetry run`. The commands you then run are the same ones you'd use if you installed your package with `pip`.

### Shell Script

Following are some simple shell scripts (e.g. `my-tap.sh`) that can save you from typing `poetry install` and `poetry run my-tap` too many times.
### uv

#### Taps

```bash
#!/bin/sh

# This simple script allows you to test your tap from any directory, while still taking
# advantage of the poetry-managed virtual environment.
# Adapted from: https://github.com/python-poetry/poetry/issues/2179#issuecomment-668815276

unset VIRTUAL_ENV

STARTDIR=$(pwd)
TOML_DIR=$(dirname "$0")

cd "$TOML_DIR" || exit
poetry install 1>&2
poetry run my-tap $*
```

#### Targets

```bash
#!/bin/sh

# This simple script allows you to test your target from any directory, while still taking
# advantage of the poetry-managed virtual environment.
# Adapted from: https://github.com/python-poetry/poetry/issues/2179#issuecomment-668815276

unset VIRTUAL_ENV

STARTDIR=$(pwd)
TOML_DIR=$(dirname "$0")

cd "$TOML_DIR" || exit
poetry install 1>&2
poetry run my-target $* < /dev/stdin
```
Prefix all your commands with `uv run`. The commands you then run are the same ones you'd use if you installed your package with `pip` or `uv pip`.

### Meltano

Expand All @@ -175,7 +293,7 @@ plugins:
### Comparison
| | native/shell/`poetry` | `meltano` |
| | native/shell/`poetry`/`uv` | `meltano` |
| ------------------- | :-----------------------------------------------------------------------------------------: | :------------------------------------------------------------------: |
| Configuration store | Config JSON file (`--config=path/to/config.json`) or environment variables (`--config=ENV`) | `meltano.yml`, `.env`, environment variables, or Meltano's system db |
| Simple invocation | `my-tap --config=...` | `meltano invoke my-tap` |
Expand Down
Loading

0 comments on commit 02cabc0

Please sign in to comment.