[YSQL][PostGIS][SQLsmith] Segmentation fault in getPostgisConstants()

[def-]

Jira Link: DB-1012

Description

To find further problems I ran SQLsmith against the postgis data created by postgis regress tests.

Can be reproduced against yugabyte-2.11.2.0 on CentOS:

select
  ref_0.id as c0,
  61 as c1,
  public.postgis_lib_build_date() as c2,
  pg_catalog.pg_reload_conf() as c3,
  ref_0.id as c4
from
  tm.geogs as ref_0
where case when case when cast(null as anyrange) < cast(null as anyrange) then cast(null as lseg) else cast(null as lseg) end
         > cast(nullif(cast(coalesce(cast(null as lseg),
          cast(nullif(cast(null as lseg),
            cast(null as lseg)) as lseg)) as lseg),
        cast(null as lseg)) as lseg) then (select public.st_asmvt(g) from public.indexempty)
       else (select public.st_asmvt(g) from public.indexempty)
       end
     <= cast(nullif(pg_catalog.timestamp_send(
      cast(cast(null as "timestamp") as "timestamp")),
    case when (EXISTS (
          select
              ref_1.g as c0,
              ref_1.id as c1
            from
              tm.geometrycollection4326 as ref_1
            where false
            limit 161))
        or (ref_0.id is not NULL) then case when ((ref_0.g is NULL)
            and ((cast(null as "timestamp") >= cast(null as timestamptz))
              or (((cast(null as "timestamp") >= cast(null as date))
                  or (ref_0.g is NULL))
                or (cast(null as "bit") = cast(null as "bit")))))
          and (cast(null as bytea) = cast(null as bytea)) then cast(nullif(cast(null as bytea),
          cast(null as bytea)) as bytea) else cast(nullif(cast(null as bytea),
          cast(null as bytea)) as bytea) end
         else case when ((ref_0.g is NULL)
            and ((cast(null as "timestamp") >= cast(null as timestamptz))
              or (((cast(null as "timestamp") >= cast(null as date))
                  or (ref_0.g is NULL))
                or (cast(null as "bit") = cast(null as "bit")))))
          and (cast(null as bytea) = cast(null as bytea)) then cast(nullif(cast(null as bytea),
          cast(null as bytea)) as bytea) else cast(nullif(cast(null as bytea),
          cast(null as bytea)) as bytea) end
         end
      ) as bytea);

Backup of the data: postgis_reg.sql.zip
backtrace from gdb:

Core was generated by `postgres: yugabyte postgis_reg 127.0.0.1(54844) SELECT                        '.
Program terminated with signal 11, Segmentation fault.
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x0000000000a440ab in MemoryContextStrdup (context=0x2448000, string=string@entry=0x0) at ../../../../../../../src/postgres/src/backend/utils/mmgr/mcxt.c:1229
#2  0x00007fb84afeb440 in getPostgisConstants () at lwgeom_pg.c:164
#3  postgis_initialize_cache () at lwgeom_pg.c:240
#4  0x00007fb84afa7f62 in pgis_asmvt_transfn (fcinfo=0x2f32188) at lwgeom_out_mvt.c:136
#5  0x00000000006ca1fa in ExecInterpExpr (state=0x2515f60, econtext=0x25145a0, isnull=<optimized out>) at ../../../../../../src/postgres/src/backend/executor/execExprInterp.c:1687
#6  0x00000000006e37f1 in ExecEvalExprSwitchContext (isNull=0x7fff836fa56f, econtext=<optimized out>, state=0x2515f60) at ../../../../../../src/postgres/src/include/executor/executor.h:321
#7  advance_aggregates (aggstate=0x2514378, aggstate=0x2514378) at ../../../../../../src/postgres/src/backend/executor/nodeAgg.c:685
#8  agg_retrieve_direct (aggstate=0x2514378) at ../../../../../../src/postgres/src/backend/executor/nodeAgg.c:2108
#9  ExecAgg (pstate=0x2514378) at ../../../../../../src/postgres/src/backend/executor/nodeAgg.c:1766
#10 0x0000000000703202 in ExecProcNode (node=0x2514378) at ../../../../../../src/postgres/src/include/executor/executor.h:249
#11 ExecSetParamPlan (node=<optimized out>, econtext=econtext@entry=0x2f33f28) at ../../../../../../src/postgres/src/backend/executor/nodeSubplan.c:1095
#12 0x00000000006c60b8 in ExecEvalParamExec (state=state@entry=0x2efd198, op=op@entry=0x2efef40, econtext=econtext@entry=0x2f33f28) at ../../../../../../src/postgres/src/backend/executor/execExprInterp.c:2298
#13 0x00000000006ca3b6 in ExecInterpExpr (state=0x2efd198, econtext=0x2f33f28, isnull=<optimized out>) at ../../../../../../src/postgres/src/backend/executor/execExprInterp.c:1016
#14 0x00000000006d941c in ExecEvalExprSwitchContext (isNull=0x7fff836fa74f, econtext=0x2f33f28, state=0x2efd198) at ../../../../../../src/postgres/src/include/executor/executor.h:321
#15 ExecQual (econtext=0x2f33f28, state=0x2efd198) at ../../../../../../src/postgres/src/include/executor/executor.h:390
#16 ExecScan (node=0x2efc038, accessMtd=0x704830 <ForeignNext>, recheckMtd=0x704920 <ForeignRecheck>) at ../../../../../../src/postgres/src/backend/executor/execScan.c:199
#17 0x00000000006ce5db in ExecProcNode (node=0x2efc038) at ../../../../../../src/postgres/src/include/executor/executor.h:249
#18 ExecutePlan (execute_once=<optimized out>, dest=0x26da708, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x2efc038, estate=0x2514118) at ../../../../../../src/postgres/src/backend/executor/execMain.c:1730
#19 standard_ExecutorRun (queryDesc=0x3fd1d18, direction=<optimized out>, count=0, execute_once=<optimized out>) at ../../../../../../src/postgres/src/backend/executor/execMain.c:367
#20 0x00007fb859b628bd in pgss_ExecutorRun (queryDesc=0x3fd1d18, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at ../../../../../src/postgres/contrib/pg_stat_statements/pg_stat_statements.c:947
#21 0x00007fb85995b3aa in ybpgm_ExecutorRun (queryDesc=0x3fd1d18, direction=ForwardScanDirection, count=0, execute_once=<optimized out>) at ../../../../../src/postgres/contrib/yb_pg_metrics/yb_pg_metrics.c:499
#22 0x000000000088564b in PortalRunSelect (portal=portal@entry=0x244e118, forward=forward@entry=true, count=0, count@entry=9223372036854775807, dest=dest@entry=0x26da708) at ../../../../../../src/postgres/src/backend/tcop/pquery.c:955
#23 0x0000000000887141 in PortalRun (portal=portal@entry=0x244e118, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true, dest=dest@entry=0x26da708, altdest=altdest@entry=0x26da708, completionTag=0x7fff836fac20 "") at ../../../../../../src/postgres/src/backend/tcop/pquery.c:784
#24 0x0000000000884ad8 in exec_simple_query (query_string=0x2282938 "select  \n  ref_0.id as c0, \n  61 as c1, \n  public.postgis_lib_build_date() as c2, \n  pg_catalog.pg_reload_conf() as c3, \n  ref_0.id as c4\nfrom \n  tm.geogs as ref_0\nwhere case when case when cast(null "..., query_string@entry=0x7fff836faba0 "\030\341D\002") at ../../../../../../src/postgres/src/backend/tcop/postgres.c:1161
#25 yb_exec_simple_query_impl (query_string=query_string@entry=0x2282938) at ../../../../../../src/postgres/src/backend/tcop/postgres.c:4434
#26 0x000000000087f8b8 in yb_exec_query_wrapper (exec_context=exec_context@entry=0x2282000, restart_data=restart_data@entry=0x7fff836fadf0, functor=functor@entry=0x884620 <yb_exec_simple_query_impl>, functor_context=functor_context@entry=0x2282938) at ../../../../../../src/postgres/src/backend/tcop/postgres.c:4419
#27 0x00000000008802ec in yb_exec_simple_query (query_string=query_string@entry=0x2282938 "select  \n  ref_0.id as c0, \n  61 as c1, \n  public.postgis_lib_build_date() as c2, \n  pg_catalog.pg_reload_conf() as c3, \n  ref_0.id as c4\nfrom \n  tm.geogs as ref_0\nwhere case when case when cast(null "..., exec_context=exec_context@entry=0x2282000) at ../../../../../../src/postgres/src/backend/tcop/postgres.c:4449
#28 0x0000000000882213 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x227dfe8, dbname=0x2303fe8 "postgis_reg", username=0x2313fe8 "yugabyte") at ../../../../../../src/postgres/src/backend/tcop/postgres.c:5084
#29 0x000000000049e292 in BackendRun (port=0x216c960) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4470
#30 BackendStartup (port=0x216c960) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4136
#31 ServerLoop () at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1754
#32 0x00000000007ea21f in PostmasterMain (argc=argc@entry=23, argv=argv@entry=0x2046000) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1417
#33 0x000000000073588a in PostgresServerProcessMain (argc=23, argv=0x2046000) at ../../../../../../src/postgres/src/backend/main/main.c:234
#34 0x0000000000735a89 in main ()

Coredump: core.10294.zip

[andrei-mart]

I'm getting different error:

yugabyte=# select
yugabyte-#   ref_0.id as c0,
yugabyte-#   61 as c1,
yugabyte-#   public.postgis_lib_build_date() as c2,
yugabyte-#   pg_catalog.pg_reload_conf() as c3,
yugabyte-#   ref_0.id as c4
yugabyte-# from
yugabyte-#   tm.geogs as ref_0
yugabyte-# where case when case when cast(null as anyrange) < cast(null as anyrange) then cast(null as lseg) else cast(null as lseg) end
yugabyte-#          > cast(nullif(cast(coalesce(cast(null as lseg),
yugabyte(#           cast(nullif(cast(null as lseg),
yugabyte(#             cast(null as lseg)) as lseg)) as lseg),
yugabyte(#         cast(null as lseg)) as lseg) then (select public.st_asmvt(g) from public.indexempty)
yugabyte-#        else (select public.st_asmvt(g) from public.indexempty)
yugabyte-#        end
yugabyte-#      <= cast(nullif(pg_catalog.timestamp_send(
yugabyte(#       cast(cast(null as "timestamp") as "timestamp")),
yugabyte(#     case when (EXISTS (
yugabyte(#           select
yugabyte(#               ref_1.g as c0,
yugabyte(#               ref_1.id as c1
yugabyte(#             from
yugabyte(#               tm.geometrycollection4326 as ref_1
yugabyte(#             where false
yugabyte(#             limit 161))
yugabyte(#         or (ref_0.id is not NULL) then case when ((ref_0.g is NULL)
yugabyte(#             and ((cast(null as "timestamp") >= cast(null as timestamptz))
yugabyte(#               or (((cast(null as "timestamp") >= cast(null as date))
yugabyte(#                   or (ref_0.g is NULL))
yugabyte(#                 or (cast(null as "bit") = cast(null as "bit")))))
yugabyte(#           and (cast(null as bytea) = cast(null as bytea)) then cast(nullif(cast(null as bytea), 
yugabyte(#           cast(null as bytea)) as bytea) else cast(nullif(cast(null as bytea),
yugabyte(#           cast(null as bytea)) as bytea) end
yugabyte(#          else case when ((ref_0.g is NULL)
yugabyte(#             and ((cast(null as "timestamp") >= cast(null as timestamptz))
yugabyte(#               or (((cast(null as "timestamp") >= cast(null as date))
yugabyte(#                   or (ref_0.g is NULL))
yugabyte(#                 or (cast(null as "bit") = cast(null as "bit")))))
yugabyte(#           and (cast(null as bytea) = cast(null as bytea)) then cast(nullif(cast(null as bytea), 
yugabyte(#           cast(null as bytea)) as bytea) else cast(nullif(cast(null as bytea),
yugabyte(#           cast(null as bytea)) as bytea) end
yugabyte(#          end
yugabyte(#       ) as bytea);
ERROR:  pgis_asmvt_transfn: parameter row cannot be other than a rowtype

Ii is hard to say why it errors out.
Chances are that I screwed up something by restoring the attached postgis_reg.sql.zip, which contained system data from different version.
However the error message comes from pgis_asmvt_transfn at lwgeom_out_mvt.c:168, which is beyond the point of the segmentation fault. I'm assuming incorrectly installed extension.

[def-]

Can still reproduce on 2.11.2.0-b89 with postgis installed as described in https://docs.yugabyte.com/latest/api/ysql/extensions/#postgis:

$ rm -rf ~/var
$ bin/yugabyted start
$ bin/ysqlsh -c "CREATE EXTENSION postgis;"
$ bin/ysqlsh < postgis_reg.sql
$ bin/ysqlsh
ysqlsh (11.2-YB-2.11.2.0-b0)
Type "help" for help.

yugabyte=# select
yugabyte-#   ref_0.id as c0,
yugabyte-#   61 as c1,
yugabyte-#   public.postgis_lib_build_date() as c2,
yugabyte-#   pg_catalog.pg_reload_conf() as c3,
yugabyte-#   ref_0.id as c4
yugabyte-# from
yugabyte-#   tm.geogs as ref_0
yugabyte-# where case when case when cast(null as anyrange) < cast(null as anyrange) then cast(null as lseg) else cast(null as lseg) end
yugabyte-#          > cast(nullif(cast(coalesce(cast(null as lseg),
yugabyte(#           cast(nullif(cast(null as lseg),
yugabyte(#             cast(null as lseg)) as lseg)) as lseg),
yugabyte(#         cast(null as lseg)) as lseg) then (select public.st_asmvt(g) from public.indexempty)
yugabyte-#        else (select public.st_asmvt(g) from public.indexempty)
yugabyte-#        end
yugabyte-#      <= cast(nullif(pg_catalog.timestamp_send(
yugabyte(#       cast(cast(null as "timestamp") as "timestamp")),
yugabyte(#     case when (EXISTS (
yugabyte(#           select
yugabyte(#               ref_1.g as c0,
yugabyte(#               ref_1.id as c1
yugabyte(#             from
yugabyte(#               tm.geometrycollection4326 as ref_1
yugabyte(#             where false
yugabyte(#             limit 161))
yugabyte(#         or (ref_0.id is not NULL) then case when ((ref_0.g is NULL)
yugabyte(#             and ((cast(null as "timestamp") >= cast(null as timestamptz))
yugabyte(#               or (((cast(null as "timestamp") >= cast(null as date))
yugabyte(#                   or (ref_0.g is NULL))
yugabyte(#                 or (cast(null as "bit") = cast(null as "bit")))))
yugabyte(#           and (cast(null as bytea) = cast(null as bytea)) then cast(nullif(cast(null as bytea),
yugabyte(#           cast(null as bytea)) as bytea) else cast(nullif(cast(null as bytea),
yugabyte(#           cast(null as bytea)) as bytea) end
yugabyte(#          else case when ((ref_0.g is NULL)
yugabyte(#             and ((cast(null as "timestamp") >= cast(null as timestamptz))
yugabyte(#               or (((cast(null as "timestamp") >= cast(null as date))
yugabyte(#                   or (ref_0.g is NULL))
yugabyte(#                 or (cast(null as "bit") = cast(null as "bit")))))
yugabyte(#           and (cast(null as bytea) = cast(null as bytea)) then cast(nullif(cast(null as bytea),
yugabyte(#           cast(null as bytea)) as bytea) else cast(nullif(cast(null as bytea),
yugabyte(#           cast(null as bytea)) as bytea) end
yugabyte(#          end
yugabyte(#       ) as bytea);
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

I'm wondering why you are not seeing any of these crashes. Is it a yugabyte, postgres or postgis version difference? I used the exact versions as described in the documentation. I think we shouldn't close these crashes before this is cleared up.