to_netcdf on Python 3: "string" qualifier on attributes

[jhamman]

I've had a number of people ask me about this and I think we can figure out a way to fix this. In python3, variabile attributes in files written with Dataset.to_netcdf end up with the "string" type qualifier shown below. This causes all sorts of problems with other netcdf programs. Is this related to Unidata/netcdf4-python#485?

PRISM$ ncdump -h prism_historical_conus4k.189501-201510.nc 
netcdf prism_historical_conus4k.189501-201510 {
dimensions:
    latitude = 621 ;
    longitude = 1405 ;
    time = 1450 ;
variables:
    double latitude(latitude) ;
    double longitude(longitude) ;
    int64 time(time) ;
        string time:units = "days since 1895-01-01 00:00:00" ;
        string time:calendar = "proleptic_gregorian" ;
    float prcp(time, latitude, longitude) ;
        string prcp:units = "mm" ;
        string prcp:description = "precipitation " ;
        string prcp:long_name = "precipitation" ;

// global attributes:
        string :title = "PRISM: Parameter-elevation Regressions on Independent Slopes Model" ;
}

cc @lizaclark

[rafa-guedes]

I had that happening with python2 as well - just for netcdf4 files though, because of the new string type I guess.. when writing as netcdf4-classic that string output was not shown.

[shoyer]

Yes, this is probably related to the netCDF4-python change discussed here: Unidata/netcdf4-python#388

Unfortunately, lots of older netCDF programs don't know how to interpret the netCDF4 "string" dtype -- and to be honest, even the netCDF-C library doesn't always properly save such data as Unicode yet, though that should be fixed in the next release (4.4.0 -- see Unidata/netcdf-c#127 for more details).

So an alternative choice would be to attempt to always encode unicode (str on Python 3) attributes as ASCII when saving with netCDF4, e.g.,

def maybe_encode(string, encoding='ascii'):
    try:
        return string.encode(encoding)
    except UnicodeEncodeError:
        return string

[jhamman]

So an alternative choice would be to attempt to always encode unicode (str on Python 3) attributes as ASCII when saving with netCDF4, e.g.,

Exactly what I have done in the past.

Does this seem like something we should change/fix?

[shoyer]

Yes, I would support this change.

On Thu, Dec 17, 2015 at 10:43 AM, Joe Hamman [email protected]
wrote:

So an alternative choice would be to attempt to always encode unicode (str
on Python 3) attributes as ASCII when saving with netCDF4, e.g.,

Exactly what I have done in the past.

Does this seem like something we should change/fix?

—
Reply to this email directly or view it on GitHub
#681 (comment).

[shoyer]

This has been fixed upstream: Unidata/netcdf4-python#533

[byersiiasa]

Is this still possibly an issue?
We have been writing out netCDFs using xarray but having trouble opening in other software, e.g. ArcGIS when using NETCDF4 format.
However, works fine when using NETCDF4_CLASSIC

[shoyer]

@byersiiasa What version of netCDF4-Python are you using? This was fixed in netCDF4-Python v1.2.3.

[byersiiasa]

Turns out to be 1.2.2!. I am using the Anaconda installation and for some reason it is not updating to the latest version.
@shoyer Many thanks for your ongoing support on xarray btw, it works well and has been fairly painless to use