Define library extras and refactor S3 readers

[javihern98]

Overview

Currently the s3 support delivers quite a lot of libraries (15) that are always installed regardless of the use case. Almost all functionalities in the library should be based on the minimal amount of libraries possible.

When attempting to install vtlengine as a dependency, the poetry dependency resolver takes a lot of time.

Task to perform

• Explore which libraries may be optional if they do not break core functionalities such as the semantic analysis and run over vtl code.
• Generate extras based on its relevance and number of libraries imported.
• Ensure a user friendly message is used whenever attempting to use a functionality that requires an extra (see https://github.com/bis-med-it/pysdmx/blob/develop/src/pysdmx/__extras_check.py)
• Ensure we can reduce the code base on S3 functionality by using only pandas (with extras)
• Edit the pyproject.toml to adjust to Poetry 2.0: https://python-poetry.org/docs/pyproject/
• Attempt to use only libraries that have as well APT packages (usually as python3-LIBRARY_NAME). If not, try to find a way to compile them using source code.

[javihern98]

Handling of files on the cloud using Pandas storage options:
https://pandas.pydata.org/pandas-docs/version/2.1/user_guide/io.html#reading-writing-remote-files

[javihern98]

Maybe instead of adding an extra on s3, we could write some installation docs that suggest the S3 compatibility and only define somehow the storage options for the engine, with the use of:

• A global config parameter
• An environment variable (preferred)
• Passing somehow some parameters for input and output storage options?

In any case, if the users wants to have the S3 compatibilty, they may always install the required extras. It takes an awful lot of time to resolve the dependencies for pandas extras aws and gcp, but once they are defined in the lock it is quite fast again. It also downloads a lot of extra libraries that are only used for s3 and are not part of the rest of the library.

[ghomem]

Some notes:

• agreed on the extras approach
• the libraries on which the VTL Engine depends should be specified as ranges, rather than fixed versions
  • the engine itself is a library and would fail to coexist with other libraries if the dependencies were specified with fixed versions
  • xbridge defines the dependencies correctly: https://github.com/Meaningful-Data/xbridge/blob/main/pyproject.toml#L29

I am attaching the full list of Ubuntu 24.04 (latest LTS Ubuntu) python libraries so that an effort to make the VTL Engine runnable on plain Ubuntu can be done.

libs-ubuntu2404.txt