update_database fails for very large object ids

[mmd-osm]

As reported here: https://gis.stackexchange.com/questions/273204/what-is-the-max-value-for-node-ids-in-overpass, someone tried to load custom data into Overpass using very large node ids. This currently fails to a large memory consumption for Random Files.

Example:

<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' generator='JOSM'>
  <node id='540000000000001' timestamp='2013-09-07T20:06:31Z' uid='136321' user='Teddy73' version='2' changeset='17717760' lat='49.3626573' lon='7.2221713'>
    <tag k='power' v='tower' />
  </node>
</osm>

[drolbr]

The deeper problem is that it leads to a large .map.idx file. This is a huge problem because the .idx files must be loaded into memory at each startup.

I will figure out a way for a redesign. But for the moment I can only offer a meaningful error message instead of a crash. Message added in 7dd05ef

[mmd-osm]

I can only offer a meaningful error message instead of a crash.

Can you please also document in this issue what the new limits are?

Update:

Max id for nodes is 4398046511103 (2^42 - 1)

[mikehwang]

I am also doing the same thing as @mmd-osm for the same purpose. I want to point out that an alternative to supporting huge integers as IDs is to make IDs signed integers rather than unsigned in Overpass. This would help achieve the goal to load custom datasets. Custom datasets could be negatives rather than super large. No sure how difficult of a change this would be.

Supporting huge signed integers would be the best of course 😄

[mmd-osm]

To be honest I don’t see much reason to go to a 500++ trillion range for custom data, let alone negative ids. We have about 5.5 billion nodes right now in OSM and anything in the 8-10+ billion range would already suffice for today’s and tomorrow’s data.