Implement writing pandas metadata and auto-setting cats/index #151
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Only copes with single index, not compound (as was the case in fastparquet so far).
wesm
reviewed
May 19, 2017
| pf = ParquetFile(fn) | ||
| assert set(pf.columns) == {'x', 'y', 'z'} | ||
| meta = json.loads(pf.key_value_metadata['pandas']) | ||
| assert meta['index_columns'] == ['z'] |
There was a problem hiding this comment.
So one deficiency for named indexes is that there may be a conflict between the index name and a column name in the DataFrame. I opened pandas-dev/pandas#16391, which is something we can add in a forward-compatible way
There was a problem hiding this comment.
We currently check against duplicate column names after reset_index, so you would get an error for this case
fastparquet/writer.py
Outdated
| @@ -671,6 +678,7 @@ def make_metadata(data, has_nulls=True, ignore_columns=[], fixed_text=None, | |||
| se.repetition_type = parquet_thrift.FieldRepetitionType.OPTIONAL | |||
| fmd.schema.append(se) | |||
| root.num_children += 1 | |||
| meta.value = json.dumps(pandas_metadata) | |||
| cats.value = json.dumps(catstruct) | |||
There was a problem hiding this comment.
Do you want to drop this one in favor of the pandas metadata?
There was a problem hiding this comment.
You are right, this should be removed. On reading I give a warning when this old version is present.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Only copes with single index, not compound (as was the case in fastparquet
so far).
See the spec