v0.39.0

v0.39.0 (2024-07-27)

• PyPI 🐍

Highlights

• The json module can now be used within stream maps

  stream_maps:
    animals:
      id: id
      data: "json.dumps({'description': description, 'verified': verified, 'views': views, 'created_at': created_at})"
      __else__: null

• The crypto dependencies PyJWT and cryptography are now optional and can be installed by requesting the jwt extra, i.e. singer-sdk[jwt]. These are only currently required for JWT authentication, but if your package depends on any of these, make sure the dependency is explicit.

• The pendulum and python-dateutil dependencies are no longer used by this library and will be removed in a future release. If your package relies on any of them, make sure the dependency is explicit.

• We're working on supporting msgspec for more performant IO, so keep an eye out for that 😉.

✨ New

• #2432 Developers can now customize the default logging configuration for their taps/targets by adding default_logging.yml to their package
• #2531 The json module is now avaiable to stream maps -- Thanks @grigi!
• #2529 Stream sync context is now available to all instances methods as a Stream.context attribute

🐛 Fixes

• #2554 Use mapped stream aliases when handling ACTIVATE_VERSION messages in the base target class
• #2526 Moved up the supported Python versions in the Markdown output of --about

⚙️ Under the Hood

• #2564 Make SQLSink a generic with a SQLConnector type parameter
• #2540 Implement abstract serialize_message for Singer writers
• #2259 Centralize JSON SerDe into helper functions -- Thanks @BuzzCutNorman!
• #2525 Make PyJWT and cryptography dependencies optional
• #2528 Moved class-level attributes to the top in REST tap template
• #2132 Limit internal usage of pendulum

📚 Documentation Improvements

• #2557 Document that get_starting_timestamp requires setting a non-null replication_key
• #2556 Reference state partitioning in stream partitioning page
• #2536 Prepare for RTD addons migration
• #2535 Added more intersphinx links to Python and Faker docs
• #2530 Explained how the request URL is generated from url_base, path and the sync context
• #2527 Updated the footer
• #2506 Fixed a typo in the stream maps docs

v0.38.0

v0.38.0 (2024-06-17)

✨ New

• #2433 Tap developers can now disable HTTP redirects
• #2426 Added an optional GitHub workflow to publish to PyPI with trusted publishers

🐛 Fixes

• #2438 Null replication values are now handled when incrementing bookmarks
• #2431 Updated cookiecutter VSCode launch.json to use debugpy
• #2421 An error message is now logged every time schema validation fails for any record

⚙️ Under the Hood

• #2455 Use parent datetime.datetime class in type conforming checks
• #2453 Change to return type of utc_now from pendulum.DateTime to datetime.datetime

📚 Documentation Improvements

• #2449 Add a short guide on defining a configuration schema
• #2436 Documented how context fields are passed to a child stream
• #2435 Using an empty list for __key_properties__ to disable a stream primary keys is now recommended as an alternative to null

v0.37.0

v0.37.0 (2024-04-29)

Highlights

• You can reference your streams with glob patterns in stream map configuration:

  stream_maps:
    # This will affect all streams with names starting with 'my_prefix_'
    "my_prefix_*":
      id: id
      first_name: first_name
      email:     # drop the PII field from RECORD and SCHEMA messages
      email_domain: email.split('@')[-1]
      email_hash: md5(config['hash_seed'] + email)
      __else__: null

✨ New

• #2389 JSON schema keyword allOf is now supported
• #1888 Added support for glob patterns in source stream names -- Thanks @DouweM!
• #2345 PropertiesList can now behave as an iterable -- Thanks @ReubenFrankel!

🐛 Fixes

• #2352 Removed unnecessary and problematic column caching -- Thanks @raulbonet!
• #2375 Added sensitive: true to password settings in templates
• #2301 Unmapped sub-fields in object-type fields are now no longer dropped when the field declares additionalProperties
• #2348 Added a condition to the No schema for record field warning -- Thanks @tobiascadee!
• #2342 Avoid failing if VSCode IDE config is not requested for target and mapper cookiecutter templates -- Thanks @ReubenFrankel!
• #2331 Allow importlib-resources >=6.3.2

⚙️ Under the Hood

• #2205 Added a jwt package extra, but the cryptography and jwt dependencies are still installed by default for now

📚 Documentation Improvements

• #2326 Documented BATCH as a default plugin capability -- Thanks @ReubenFrankel!

v0.36.1

v0.36.1 (2024-03-13)

🐛 Fixes

• #2310 Limited importlib-resources to < 6.2 due to breaking changes
• #2288 Bumped min joblib dependency to >=1.3.0 and replaced deprecated parallel_backend with parallel_config -- Thanks @BuzzCutNorman!
• #2281 The state_partition_context dictionary is now correctly interpolated in the error message when duplicate partitions/contexts are detected in the input state
• #2274 Test workflow now fails for any Python matrix job failure in cookiecutter template -- Thanks @ReubenFrankel!

v0.36.0

v0.36.0 (2024-02-26)

✨ New

• #2241 JSON schema keywords such as maxLength are now supported in StringType, IntegerType and NumberType JSON schema helpers
• #2263 Nested settings are now documented in the output of --about --format=markdown
• #2248 Targets now accept a batch_size_rows setting to configure how many rows are loaded in each record batch -- Thanks @BuzzCutNorman!

🐛 Fixes

• #2258 Database disconnects are now handled via SQLAlchemy pool_pre_ping parameter

⚙️ Under the Hood

• #2220 Deprecated singer_sdk.authenticators.BasicAuthenticator in favor of requests.auth.HTTPBasicAuth

v0.35.2

v0.35.2 (2024-02-19)

🐛 Fixes

• #2252 Null type is now correctly appended as {"type": "null"} to oneOf types

v0.35.1

v0.35.1 (2024-02-15)

🐛 Fixes

• #2243 Flattening level of record is now forced to match the provided flattened schema -- Thanks @joaopamaral!
• #2245 Instances of oneOf are now handled by null-appending logic
• #2242 Hard and soft deletes for handling ACTIVATE_VERSION messages now use the same WHERE clause -- Thanks @vitoravancini!
• #2232 Test workflow job now fails for unsupported Python versions in cookiecutter templates -- Thanks @ReubenFrankel!
• #2225 SQL columns that are non-nullable but not required (i.e. not part of a primary key) are now not included in the "required": [...] array of the discovered JSON schema

📚 Documentation Improvements

• #2239 Linked reference docs to source code
• #2231 Added an example implemetation of JSON schema validation that uses fastjsonschema
• #2219 Added reference docs for tap & target testing helpers

v0.35.0

v0.35.0 (2024-02-02)

Highlights

• The package now acts a proper pytest plugin, so you may need to remove pytest_plugins = ("singer_sdk.testing.pytest_plugin",) from your tests/conftest.py file for test fixtures to be loaded correctly.
• Dropped support for Python 3.7 (EOL 2023-06-27).
• Dropped direct dependency on pytz. If your package uses pytz in any capacity, you may need to add it as an explicit dependency to your project.
• Stream maps can now generate fake data! See the docs for more details.

✨ New

• #2208 Allow users to disable schema validation in targets
• #2170 Generate fake data with stream maps -- Thanks @ReubenFrankel!
• #937 Support validating configuration for any tap with a dynamic catalog
• #2144 Support fanning out parent record into multiple child contexts/syncs
• #1918 End RESTStream pagination if an empty page is received

🐛 Fixes

• #2203 Fix serialization of arbitrary objects (e.g. ObjectId from mongoDB) during flattening -- Thanks @dgawlowsky!
• #2200 Quote column names in INSERT statement
• #2195 Include empty schemas directory in REST tap cookiecutter
• #2187 Replace use of deprecated jsonschema._RefResolver with recommended referencing library
• #2184 Reduce amount of unnecessary whitespace in Singer output
• #2183 Ensure .vscode directory is included when requested in cookiecutters and avoid failing if it does not exist
• #2180 Limit supported Python versions in --about output to existing ones
• #2108 Log sink name when an unhandled error occurs during setup
• #2158 Fix pytest plugin declaration so it can be used without requiring defining pytest_plugins in conftest.py
• #2105 Default handling of ACTIVATE_VERSION messages to soft deletes and add new SQLConnector.delete_old_versions method

⚙️ Under the Hood

• #2189 Use functools.lru_cache instead of the stale memoization library (#1981)
• #2188 Remove unused logger parameter from private catalog helper functions
• #2143 Drop support for Python 3.7
• #2157 Remove pytz dependency and use datetime.timezone.utc instead of pytz.UTC where possible
• #2136 Create interface for schema validation in sinks, and implement it for python-jsonschema -- Thanks @BuzzCutNorman!
• #2130 Allow loading stream schemas from importlib.resources.abc.Traversable types

📚 Documentation Improvements

• #2204 Document supported package extras
• #2186 Call out minimum recommended cookiecutter version
• #2168 Explain Progress is not resumable if interrupted in docs FAQ
• #2140 Update auth caching example to use functools.cached_property

v0.34.1

v0.34.1 (2023-12-19)

🐛 Fixes

• #2118 Output JSONPath expression with match count message -- Thanks @mjsqu!
• #2107 Respect forced replication method when retrieving state
• #2094 Use nulls_first when available to order NULL results in incremental SQL streams

⚙️ Under the Hood

• #1733 Test with Python 3.12 🐍
• #2095 Use CursorResult.mappings() in SQL streams
• #2092 Use datetime.fromisoformat in other places
• #2090 Explicitly use T iso date separator

📚 Documentation Improvements

• #2111 Fix broken requests documentation links -- Thanks @mjsqu!

v0.34.0

v0.34.0 (2023-12-05)

Hightlights

• Date-time parsing in target sinks is now significantly faster, thanks to @BuzzCutNorman!
• Experimental support for Parquet encoding in BATCH messages, thanks to @jamielxcarter!

✨ New

• #2044 Add Parquet as a batch encoding option -- Thanks @jamielxcarter!
• #768 Better error messages when config validation fails
• #1854 Make stream logger a child of the tap logger

🐛 Fixes

• #2076 Make a explicit dependency on python-dateutil

⚙️ Under the Hood

• #2070 Parse dates with datetime.fromisoformat/backports.datetime_fromisoformat -- Thanks @BuzzCutNorman!
• #2056 Break up TapTestClassFactory._annotate_test_class into simpler methods
• #2058 Implement a SingerWriter class in singer_sdk.io_base and use it to emit Singer messages -- Thanks @BuzzCutNorman!
• #2061 Simplify target template file names with post_gen_project.py hook -- Thanks @vicmattos!
• #2060 Simplify tap template file names with post_gen_project.py hook -- Thanks @vicmattos!

📚 Documentation Improvements

• #2039 Add 404 page with sphinx-notfound-page
• #2037 Add flattening configuration examples

Full Changelog: v0.33.1...v0.34.0