osm2pgsql: stuck lease during data import involving DROP TABLE

[otan]

We currently do this with four tables at the same table, non-transactionally:

• DROP TABLE IF NOT EXISTS planet_osm_point
• DROP TABLE IF NOT EXISTS planet_osm_point_tmp
• CREATE TABLE planet_osm_point ( .... )
• COPY in data
• CREATE TABLE planet_osm_point_tmp AS SELECT * FROM planet_osm_point WHERE ST_IsValid(way) ORDER BY way
• DROP TABLE planet_osm_point

The final DROP gets stuck for 5 minutes. I've noticed this fails even with concurrency of the tool I'm running set to one, but it doesn't seem to fail using the cockroach sql shell....

Reproduction steps:

• Checkout osm2pgsql-dev/osm2pgsql@master...otan-cockroach:crdb_branch, build using cd build && cmake .. && make && cd ..
• Run cockroach single node
• Download the dataset: https://download.geofabrik.de/europe/iceland-latest.osm.pbf
• Run the tool ./build/osm2pgsql -U root -d defaultdb -H localhost -P 26257 --number-processes 1 -C 20480 ~/iceland-latest.osm.pbf

The SHOW JOBS appears as following:

        job_id       |     job_type     |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | statement | user_name |  status   | running_status |             created              |             started              |             finished             |             modified             | fraction_completed | error | coordinator_id
---------------------+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+-----------+----------------+----------------------------------+----------------------------------+----------------------------------+----------------------------------+--------------------+-------+-----------------
  576613482874667009 | SCHEMA CHANGE    | DROP TABLE defaultdb.public.planet_osm_polygon                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |           | root      | running   | NULL           | 2020-07-29 16:07:50.113896+00:00 | 2020-07-29 16:07:50.227218+00:00 | NULL                             | 2020-07-29 16:07:50.113896+00:00 |                  0 |       |              1
....
  576613268613726209 | SCHEMA CHANGE GC | GC for DROP TABLE IF EXISTS defaultdb.public.planet_osm_polygon CASCADE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |           | root      | running   | NULL           | 2020-07-29 16:06:44.728305+00:00 | 2020-07-29 16:06:44.780214+00:00 | NULL                             | 2020-07-29 16:06:44.815253+00:00 |                  0 |       |              1

(The GC job probably exists because I created and dropped an earlier table).

Here is a dump I have of the logs from cockroach start-single-node from a clean cockroach-data directory: https://gist.github.com/0475e3348605fd59ef2bb9e1d5de6a60. There seems to be a lot of statistic changes, wonder if that's part of the problem

[ajwerner]

These symptoms indicate that somewhere we're holding on to a lease for planet_osm_point. @rohany indicated that the hack here might be causing problems:

cockroach/pkg/sql/schema_changer.go

Lines 220 to 227 in 27f8f1e

	// TODO (rohany): This is a hack and should be removed once #51865 lands.
	// In case we used this planner to access any descriptors, we need to
	// release any leases we might have acquired during planning and execution.
	defer func() { localPlanner.Descriptors().ReleaseAll(ctx) }()
	stmt, err := parser.ParseOne(table.CreateQuery)
	if err != nil {
	return err
	}

[otan]

I tried with that line removed, didn't seem to fix it.

[ajwerner]

Well this one turned out to be fun. We apparently have this thing called the copyMachine to deal with the COPY protocol that I know nothing about that apparently is getting used here. Anyway, it sidesteps the usual connExecutor state machine but didn't clean up after itself.

[otan]

thanks!!!