8.0.20 (21 March 2025)

Patch Release

There are no code changes in this release.

A package maintainer at Debian raised an issue with the now
outdated cx_Freeze dependency in setup.py. As cx_Freeze is now only
used in the Docker build, setup.py has been removed to avoid the unneeded
dependency. Docker and package builds were tested to ensure that the changes work
as expected before this release.

Changes

• Move cx_Freeze configuration from setup.py to pyproject.toml.
  The configuration will be read during the Docker image build but not otherwise
  result in cx_Freeze being a dependency.
• Change how cx_Freeze is invoked in Dockerfile, removing the need for
  setup.py.
• Remove setup.py as it is no longer needed for package building.
• Use es_client==8.17.4 which fixes a logging issue.
• Fix copyright dates in docs/index.rst
• Update asciidoc files to reflect the new version numbers.

8.0.19 (5 March 2025)

Announcement

The ability to include hidden indices is now available in Curator. This is now
an option you can include in the options section of your configuration file.

    options:
      include_hidden: True

This will allow Curator to include hidden indices in its actions. This will not
suddenly reveal system indices, but it will allow you to include indices that
are hidden, such as indices backing certain data_streams.

This is also an option flag for the CLI Singletons, e.g. --include-hidden,
with the default value being the equivalent of --no-include_hidden.

There's an odd caveat to this, however, and it is probaby a bug in Elasticsearch.
If you have an index that starts with a dot, e.g. .my_index, and you set your
multi-target search pattern to also start with a dot, e.g. .my_*, and you
set the index settings for .my_index to be hidden: true, the index will
not be excluded from the search results when the expand_wildcards parameter
is set to include hidden indices, e.g. open,closed,hidden.

This is a bug in Elasticsearch, and not in Curator. I've reported it to the
Elasticsearch team at elastic/elasticsearch#124167,
but I'm not sure when it will be addressed. In the meantime, if you need to
guarantee hidden indices stay excluded, use search_pattern and filters
to exclude anything that needs to stay excluded.

All tests pass on versions 7.17.25 and 8.17.2 of Elasticsearch.

8.0.18 (27 February 2025)

Announcement

Release 8.0.18 allows Curator v8 to work with Curator v7.14.x and later. All tests pass for v7.14.0, v7.14.2, v7.17.7, v7.17.25, v7.17.27. Due to JVM issues with M-series processors (OpenJDK 16 < FAIL < OpenJDK 19 ), I'm unable to test v7.15.x - v7.17.6 on my MacBook Pro, but I have no reason to believe they won't as there are no API changes between the 7.14 and 7.17 series that would affect Curator in any way.

Hopefully this is helpful for those who are using Elasticsearch 7 still, and would like the improvements in Curator v8. Do note that while the action files used in Curator v7 will work with Curator v8, the client configuration files will not. There are a few differences in syntax that are in the documentation.

See https://www.elastic.co/guide/en/elasticsearch/client/curator/current/configfile.html for more information.

Changes

• Huge number of comment and code line break changes to be more PEP8 compliant.
  • This includes updates to Integration tests as well
• Update to use es_client==8.17.2, which enables Curator v8 to work with Elasticsearch v7.14.x and later.
• Add VERSION_MAX and VERSION_MIN to curator.defaults to allow for version compatibility checks. VERSION_MIN is set to 7.14.0.
• Exclude system indices by default. The list of patterns to exclude is in curator.defaults, and is presently

       EXCLUDE_SYSTEM = (
           '-.kibana*,-.security*,-.watch*,-.triggered_watch*,'
           '-.ml*,-.geoip_databases*,-.logstash*,-.tasks*'
       )

• Restored and fixed datemath integration tests for patterns that include colons. This was a problem in the past due to Elasticsearch evaluating colon characters as potential indicators of remote indices. This was fixed in version 8.7.0 of Elasticsearch.

Bugfix

All of the testing for this release revealed a shortcoming in the snapshot restore action class. It was relying on the user to provide the exact index names to restore. The restore API call to Elasticsearch allows for multi-target syntax to select and de-select indices by comma-separated patterns. While not expressly allowed, it was possible to use. The problem was that indices could not be properly matched and verified if patterns rather than exact names were used. Three functions were added to helpers.utils to make this work properly: multitarget_match, multitarget_fix, and regex_loop. The Restore class now calls multitarget_match in the _get_expected_output method to get the names from the snapshot list object. This now works with patterns, and the tests have been updated to ensure this.

8.0.17 (25 October 2024)

Bugfix

• Reported in #1727 (and I'm relieved that nobody got bit by this sooner), A
  serious bug was found where if using the age filter while deriving the
  age from the index name, it was erroneously ignoring any index that did not
  have a matching timestring pattern, which would leave a default epoch
  time of 0, which would definitely be older than your cutoff date! As a
  result, indices were matched and deleted that should not have been.

  The fix is to remove indices that do not match the pattern in the
  _get_name_based_ages method. The patch is in this release, as are updated
  tests to replicate the failure scenario. Hat tip to @giom-l for reporting this.

Changes

• Update to use es_client==8.15.2.
• Update data node detection to include data, data_content, data_hot,
  and data_warm for shrink action. This was first raised in #1621, but
  needed to go further than just adding data_hot. Hat tip to @gnobironts for
  the original pull request.
• Add docker_test/.env to .gitignore
• More formatting changes as suggested by pylint
• Improve API calls to node.info and node.stats to use filter_path

8.0.16 (6 August 2024)

Changes

• Update to use es_client==8.14.2
• Formatting changes and improvements
• Update CLI to get client using ctx.obj['configdict'] as it's already built
  by es_client.

Bugfixes

• Fix improper log levels erroneously left in from debugging. Thanks to
  @boutetnico in #1714
• es_client version 8.14.2 addresses a problem where Python 3.8 is not officially supported
  for use with voluptuous greater than 0.14.2.

8.0.15 (10 April 2024)

Announcement

• Python 3.12 support becomes official. A few changes were necessary to datetime calls
  which were still using naive timestamps. Tests across all minor Python versions from 3.8 - 3.12
  verify everything is working as expected with regards to those changes. Note that Docker builds
  are still running Python 3.11 as cx_Freeze still does not officially support Python 3.12.
• Added infrastructure to test multiple versions of Python against the code base. This requires
  you to run:
  * pip install -U hatch hatchling -- Install prerequisites
  * hatch run docker:create X.Y.Z -- where X.Y.Z is an ES version on Docker Hub
  * hatch run test:pytest -- Run the test suite for each supported version of Python
  * hatch run docker:destroy -- Cleanup the Docker containers created in docker:create

Bugfix

• A bug reported in es_client with Python versions 3.8 and 3.9 has been addressed. Going
  forward, testing protocol will be to ensure that Curator works with all supported versions of
  Python, or support will be removed (when 3.8 is EOL, for example).

Changes

• Address deprecation warning in get_alias() call by limiting indices to only open and
  closed indices via expand_wildcards=['open', 'closed'].
• Address test warnings for an improperly escaped \d in a docstring in indexlist.py
• Updated Python version in Docker build. See Dockerfile for more information.
• Docker test scripts updated to make Hatch matrix testing easier (.env file)

8.0.14 (2 April 2024)

Announcement

• A long awaited feature has been added, stealthily. It's fully in the documentation, but I do
  not yet plan to make a big announcement about it. In actions that search through indices, you
  can now specify a search_pattern to limit the number of indices that will be filtered. If
  no search pattern is specified, the behavior will be the same as it ever was: it will search
  through _all indices. The actions that support this option are: allocation, close,
  cold2frozen, delete_indices, forcemerge, index_settings, open, replicas, shrink, and snapshot.

Bugfix

• A mixup with naming conventions from the PII redacter tool got in the way of the cold2frozen
  action completing properly.

Changes

• Version bump: es_client==8.13.0
  • With the version bump to es_client comes a necessary change to calls to create a
    repository. In elastic/elasticsearch-specification#2255 it became
    clear that using type and settings as it has been was insufficient for repository
    settings, so we go back to using a request body as in older times. This change affects
    esrepomgr in one place, and otherwise only in snapshot/restore testing.
• Added the curator.helpers.getters.meta_getter to reduce near duplicate functions.
• Changed curator.helpers.getters.get_indices to use the _cat API to pull indices. The primary
  driver for this is that it avoids pulling in the full mapping and index settings when all we
  really need to return is a list of index names. This should help keep memory from ballooning
  quite as much. The function also now allows for a search_pattern kwarg to search only for
  indices matching a pattern. This will also potentially make the initial index return list much
  smaller, and the list of indices needing to be filtered that much smaller.
• Tests were added to ensure that the changes for get_indices work everywhere.
• Tests were added to ensure that the new search_pattern did not break anything, and does
  behave as expected.

8.0.13 (26 March 2024)

Bugfix

• An issue was discovered in es_client that caused default values from command-line options
  which should not have been set to override settings in configuration files.
  es_client==8.12.9 corrects this. Reported in #1708, hat tip to @rgaduput for reporting this
  bug.

8.0.12 (20 March 2024)

Bugfix

• six dependency erroneously removed from es_client. It's back in es_client==8.12.8

8.0.11 (20 March 2024)

Announcement

• With the advent of es_client==8.12.5, environment variables can now be used to automatically
  populate command-line options. The ESCLIENT_ prefix just needs to prepend the capitalized
  option name, and any hyphens need to be replaced by underscores. --http-compress True is
  automatically settable by having ESCLIENT_HTTP_COMPRESS=1. Boolean values are 1, 0, True,
  or False (case-insensitive). Options like hosts which can have multiple values just need to
  have whitespace between the values, e.g.
  ESCLIENT_HOSTS='http://127.0.0.1:9200 http://localhost:9200'. It splits perfectly. This is
  tremendous news for the containerization/k8s community. You won't have to have all of the
  options spelled out any more. Just have the environment variables assigned.
• Also, log blacklisting has made it to the command-line as well. It similarly can be set via
  environment variable, e.g. ESCLIENT_BLACKLIST='elastic_transport urllib3', or by multiple
  --blacklist entries at the command line.
• es_client has simplified things such that I can clean up arg sprawl in the command line
  scripts.

Changes

Lots of pending pull requests have been merged. Thank you to the community
members who took the time to contribute to Curator's code.

• DOCFIX - Update date math section to use y instead of Y (#1510)
• DOCFIX - Update period filtertype description (#1550)
• add .dockerignore to increase build speed (#1604)
• DOCFIX - clarification on prefix and suffix kinds (#1558)
  The provided documentation was adapted and edited.
• Use builtin unittest.mock (#1695)
  • Had to also update helpers.testers.verify_client_object.
• Display proper error when mapping incorrect (#1526) - @namreg
  Also assisting with this is @alexhornblake in #1537
  Apologies for needing to adapt the code manually since it's been so long.
• Version bumps:
  • es_client==8.12.6