Expose GDALs metadata

[rbuffat]

@sgillies This PR looks a bit messy as it includes some CI/tests updates from maint-1.8, as the current maint-1.9 tests are failing. If you have time it would be nice if you could merge maint-1.8 into maint-1.9. I could create a PR, but then I think the history of the different commits will be lost.

This PR includes the meta module of #909
This module can be used to query metadata items of GDAL https://github.com/rbuffat/Fiona/blob/add_metadata_1_9/fiona/meta.py:

E.g.:

class MetadataItem:
    # since GDAL 2.0
    CREATION_FIELD_DATA_TYPES = "DMD_CREATIONFIELDDATATYPES"
    # since GDAL 2.3
    CREATION_FIELD_DATA_SUB_TYPES = "DMD_CREATIONFIELDDATASUBTYPES"
    CREATION_OPTION_LIST = "DMD_CREATIONOPTIONLIST"
    LAYER_CREATION_OPTION_LIST = "DS_LAYER_CREATIONOPTIONLIST"
    # since GDAL 2.0
    DATASET_OPEN_OPTIONS = "DMD_OPENOPTIONLIST"
    # since GDAL 2.0
    EXTENSIONS = "DMD_EXTENSIONS"
    EXTENSION = "DMD_EXTENSION"
    VIRTUAL_IO = "DCAP_VIRTUALIO"
    # since GDAL 2.0
    NOT_NULL_FIELDS = "DCAP_NOTNULL_FIELDS"
    # since gdal 2.3
    NOT_NULL_GEOMETRY_FIELDS = "DCAP_NOTNULL_GEOMFIELDS"
    # since GDAL 3.2
    UNIQUE_FIELDS = "DCAP_UNIQUE_FIELDS"
    # since GDAL 2.0
    DEFAULT_FIELDS = "DCAP_DEFAULT_FIELDS"
    OPEN = "DCAP_OPEN"
    CREATE = "DCAP_CREATE"

It includes also a method print driver options in a user-friendly form. The advantage compared to the driver page of GDAL's homepage is as this output represents the capabilities of the local GDAL instance whereas GDAL homepages represent the driver capabilities of the latest version of GDAL. GDAL did not include tests for the XML before OSGeo/gdal@f447a31, thus the meta module contains also some GDAL XML fixes.

@jorisvandenbossche In geopandas/geopandas#1609 (comment), you mentioned that this functionality could be used for Geopandas. If you could have a look if the API would be useful for you.

>>> import fiona.meta
>>> fiona.meta.print_driver_options("GeoJSON")
Dataset Open Options:
        FLATTEN_NESTED_ATTRIBUTES:
                Description: Whether to recursively explore nested objects and produce flatten OGR attributes
                Type: boolean
                Default value: NO
        NESTED_ATTRIBUTE_SEPARATOR:
                Description: Separator between components of nested attributes
                Type: string
                Default value: _
        FEATURE_SERVER_PAGING:
                Description: Whether to automatically scroll through results with a ArcGIS Feature Service endpoint
                Type: boolean
        NATIVE_DATA:
                Description: Whether to store the native JSon representation at FeatureCollection and Feature level
                Type: boolean
                Default value: NO
        ARRAY_AS_STRING:
                Description: Whether to expose JSon arrays of strings, integers or reals as a OGR String
                Type: boolean
                Default value: NO
        DATE_AS_STRING:
                Description: Whether to expose date/time/date-time content using dedicated OGR date/time/date-time types or as a OGR String
                Type: boolean
                Default value: NO

Dataset Creation Options:
        No options available.

Layer Creation Options:
        WRITE_BBOX:
                Description: whether to write a bbox property with the bounding box of the geometries at the feature and feature collection level
                Type: boolean
                Default value: NO
        COORDINATE_PRECISION:
                Description: Number of decimal for coordinates. Default is 15 for GJ2008 and 7 for RFC7946
                Type: int
        SIGNIFICANT_FIGURES:
                Description: Number of significant figures for floating-point values
                Type: int
                Default value: 17
        NATIVE_DATA:
                Description: FeatureCollection level elements.
                Type: string
        NATIVE_MEDIA_TYPE:
                Description: Format of NATIVE_DATA. Must be "application/vnd.geo+json", otherwise NATIVE_DATA will be ignored.
                Type: string
        RFC7946:
                Description: Whether to use RFC 7946 standard. Otherwise GeoJSON 2008 initial version will be used
                Type: boolean
                Default value: NO
        WRITE_NAME:
                Description: Whether to write a "name" property at feature collection level with layer name
                Type: boolean
                Default value: YES
        DESCRIPTION:
                Description: (Long) description to write in a "description" property at feature collection level
                Type: string
        ID_FIELD:
                Description: Name of the source field that must be used as the id member of Feature features
                Type: string
        ID_TYPE:
                Description: Type of the id member of Feature features
                Type: string-select
                Accepted values: AUTO,String,Integer
        ID_GENERATE:
                Description: Auto-generate feature ids
                Type: boolean
        WRITE_NON_FINITE_VALUES:
                Description: Whether to write NaN / Infinity values
                Type: boolean
                Default value: NO

[snowman2]

This looks like it will be very useful 👍

[jorisvandenbossche]

Thanks! This indeed looks useful for the driver inference (either for #948 or for doing something similar in geopandas). And I think exposing this is useful in general, as you say, to be able to inspect the actual available drivers.

[sgillies]

Indeed, the state of the Fiona branches is not great right now. I started a merge of maint-1.8 to maint-1.9 but had to pause before I finished. It's the work of an entire afternoon or more and we'll have to tolerate some mess for a while.

@rbuffat I think that not fixing GDAL's XML is also an option for us. We can patch GDAL for the wheels we distribute or update to GDAL 3.1.1. Conda-forge can do the same.

I'm in favor of this. Let's keep it as minimal as we can. It would be great if Fiona made better UIs possible, but we shouldn't become a framework for UIs or anything like that.

[rbuffat]

I think that not fixing GDAL's XML is also an option for us.

That is a good idea. I removed them and adapted the tests for this part to only run with >= Gdal 3.1.

I'm in favor of this. Let's keep it as minimal as we can. It would be great if Fiona made better UIs possible, but we shouldn't become a framework for UIs or anything like that.

In #909 the idea was to use the driver create / layer create / open options to filter the respecting options in fiona, as we currently don't have a way to know for an option that is passed to fiona.open for which of these operations the option is meant. But as there were so many XML errors I think this has the chance to open a can of worms. The print driver options are only a byproduct of this. DCAP_VIRTUALIO and the extensions are useful for MemoryFile.

Indeed, the state of the Fiona branches is not great right now. I started a merge of maint-1.8 to maint-1.9 but had to pause before I finished. It's the work of an entire afternoon or more and we'll have to tolerate some mess for a while.

I suspect it is easier to merge maint-1.8 into 1.9 before much more is committed to 1.9. Thus I would suggest waiting with this PR until this is merged. I will also be on staycation for the next 2 weeks, thus there is no hurry.

[coveralls]

Coverage increased (+1.0%) to 87.629% when pulling e49c1d8 on rbuffat:add_metadata_1_9 into 75209bb on Toblerity:maint-1.9.

[rbuffat]

@sgillies This PR is ready to be reviewed. Note the changes to travis and appveyor. Gdal 3.1 was missing in travis and the gdal versions removed from travis were still present in appveyor. Also Python 3.9-dev was added to travis as python 3.9 should be released soon.

[sgillies]

@rbuffat this is great work. Developers downstream are looking forward to it and that's always a good sign. About the module name: would "drivers" be more clear than "meta"?

[rbuffat]

You are right, the metadata is only about driver capabilities. We have already a drvsupport module, would this also be an option?
For me, either meta, drivers or drvsupport is fine. You can decide.

[rbuffat]

As this functionality could also be useful in rasterio (not that it has to be integrated), it might be good to think for an API that works for both projects.

[sgillies]

@rbuffat I'm going to merge this and then rename the module to fiona.drivers to match the drivers module in rasterio.

[sgillies]

I was too hasty about renaming. We'll need to do that in 2.0 since we still have a drivers() in fiona.__init__.py. See 137853e.