bug: confused watch keys lead to accidents

[tzssangglass]

Issue description

with enable_admin: false, apisix will watch /apisix/plugins. see:

apisix/apisix/plugin.lua

Lines 512 to 513 in 03a1c76

	if local_conf and not local_conf.apisix.enable_admin then
	init_plugins_syncer()

If a new apisix instance connects to the same etcd cluster, the new instance will do something to /apisix/plugins/ during the init etcd phase, causing the apisix which watching /apisix/plugins to get an incorrect response.

Environment

• apisix version (cmd: apisix version): master branch
• OS (cmd: uname -a):Darwin Kernel Version 20.6.0
• OpenResty / Nginx version (cmd: nginx -V or openresty -V): apisix-base
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):3.5.0
• apisix-dashboard version, if have:
• the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
• luarocks version, if the issue is about installation (cmd: luarocks --version):3.4

Steps to reproduce

1. prepare apisix instance A, change enable_admin: false in config.yaml, and start apisix instance A
2. prepare apisix instance B, change enable_admin: false in config.yaml, and start apisix instance B
3. look at the error.log for apisix instance A, there has some error level logs

Actual result

has error logs

Error log

I've added logging to apisix instance A the line below here:

apisix/apisix/plugin.lua

Line 227 in 03a1c76

function _M.load(config)

to print the config

there error logs as below:

2021/10/09 16:52:52 [warn] 29391#6632203: *79731 [lua] plugin.lua:228: load(): config : {
  createdIndex = 1075,
  key = "/apisix/plugins/",
  modifiedIndex = 1239
}, context: ngx.timer
2021/10/09 16:52:52 [error] 29389#6632201: *79873 [lua] config_etcd.lua:591: failed to fetch data from etcd: /usr/local/Cellar/apisix/apisix/plugin.lua:249: bad argument #1 to 'ipairs' (table expected, got nil)
stack traceback:
	[C]: in function 'ipairs'
	/usr/local/Cellar/apisix/apisix/plugin.lua:249: in function 'load'
	/usr/local/Cellar/apisix/apisix/plugin.lua:496: in function 'filter'
	/usr/local/Cellar/apisix/apisix/core/config_etcd.lua:462: in function 'sync_data'
	/usr/local/Cellar/apisix/apisix/core/config_etcd.lua:547: in function </usr/local/Cellar/apisix/apisix/core/config_etcd.lua:537>
	[C]: in function 'xpcall'
	/usr/local/Cellar/apisix/apisix/core/config_etcd.lua:537: in function </usr/local/Cellar/apisix/apisix/core/config_etcd.lua:516>,  etcd key: /apisix/plugins, context: ngx.timer
2021/10/09 16:52:52 [error] 29387#6632199: *79736 [lua] config_etcd.lua:591: failed to fetch data from etcd: /usr/local/Cellar/apisix/apisix/plugin.lua:249: bad argument #1 to 'ipairs' (table expected, got nil)

Expected result

The error log output for instance A is related to the start of instance B.

And as you can see from the error log output, instance A is listening to the key /apisix/plugins/, It should not actually listen to this key. It should be listening to /apisix/plugins and the corresponding value should be

{
  clean_handlers = {},
  createdIndex = 1091,
  key = "/apisix/plugins",
  modifiedIndex = 1091,
  value = { {
      name = "api-breaker"
    }, {
      name = "authz-keycloak"
    }, {
      name = "basic-auth"
    }, {
      name = "batch-requests"
    }, {
      name = "consumer-restriction"
    }, {
      name = "cors"
    }, {
      name = "echo"
    }, {
      name = "fault-injection"
    }, {
      name = "grpc-transcode"
    }, {
      name = "hmac-auth"
    }, {
      name = "http-logger"
    }, {
      name = "ip-restriction"
    }, {
      name = "jwt-auth"
    }, {
      name = "kafka-logger"
    }, {
      name = "key-auth"
    }, {
      name = "limit-conn"
    }, {
      name = "limit-count"
    }, {
      name = "limit-req"
    }, {
      name = "openid-connect"
    }, {
      name = "prometheus"
    }, {
      name = "proxy-cache"
    }, {
      name = "proxy-mirror"
    }, {
      name = "proxy-rewrite"
    }, {
      name = "redirect"
    }, {
      name = "referer-restriction"
    }, {
      name = "request-id"
    }, {
      name = "request-validation"
    }, {
      name = "response-rewrite"
    }, {
      name = "serverless-post-function"
    }, {
      name = "serverless-pre-function"
    }, {
      name = "sls-logger"
    }, {
      name = "syslog"
    }, {
      name = "tcp-logger"
    }, {
      name = "udp-logger"
    }, {
      name = "uri-blocker"
    }, {
      name = "wolf-rbac"
    }, {
      name = "zipkin"
    }, {
      name = "server-info"
    }, {
      name = "traffic-split"
    }, {
      name = "mqtt-proxy",
      stream = true
    } }
}

[spacewander]

DP side needs to sync the plugin from the admin, if have. I am curious why instance B will write to the /apisix/plugins, as only Admin API can write to it.

[tzssangglass]

DP side needs to sync the plugin from the admin, if have. I am curious why instance B will write to the /apisix/plugins, as only Admin API can write to it.

instance B doesn't write to /apisix/plugins, it writes to /apisix/plugins/, but instance A listens for changes to /apisix/plugins/ when it listen /apisix/plugins.