Remove source_id field
#383
Comments
orangejulius
added a commit
to pelias/api
that referenced
this issue
Oct 4, 2019
This change implements the 5th and final step in the GID migration plan outlined in pelias/pelias#672 (comment). It adjusts the expectations of the API so that it requires the full GID, rather than the source_id, to be stored in the `_id` field in Elasticsearch. In the process, the `source_id` field is also no longer used in `helper/geojsonify.js`, making this the first step towards removing the `source_id` field (pelias/schema#383). Connects pelias/pelias#672
missinglink
pushed a commit
to pelias/api
that referenced
this issue
Oct 9, 2019
This change implements the 5th and final step in the GID migration plan outlined in pelias/pelias#672 (comment). It adjusts the expectations of the API so that it requires the full GID, rather than the source_id, to be stored in the `_id` field in Elasticsearch. In the process, the `source_id` field is also no longer used in `helper/geojsonify.js`, making this the first step towards removing the `source_id` field (pelias/schema#383). Connects pelias/pelias#672
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Each of our Elasticsearch documents stores the original source ID from the upstream data in two places: as part of the GID in the
_idfield, and in thesource_idfield.Conservatively estimating 10 bytes per record, this probably represents about 6GB of duplicated data in a full planet build, and thus it would make sense to get rid of it when possible.
There will be changes required to the API, pelias/model, and here in pelias/schema to do this.
We'll also have to ensure reasonable backwards compatibility is maintained. Especially in the API, where the
source_idfield in our GeoJSON responses will still be required.The text was updated successfully, but these errors were encountered: