Getting an error "starting instance: nil"

[YegorMedvedev]

Summary

I have a Golang plugin, which handles a custom API authentication. I'm also using Kong-ingress in Kubernetes in order to deploy it.

At some point, w/o any obvious reasons I'm getting an error starting instance: nil and it totally breaks Kong and the only way to fix it is to restart kong-ingress pod.

Please find logs below.

Additional Details & Logs

• Kong version (kong:alpine)

2020/10/05 14:39:16 [error] 29#0: *10525438 [kong] go.lua:429 [jwt-b2c] starting instance: nil, client: ****, server: kong, request: "GET /accounts HTTP/1.1", host: "****"
2020/10/05 14:39:16 [error] 29#0: *10525438 lua coroutine: runtime error: unknown reason
stack traceback:
coroutine 0:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:432: in function 'reset_and_get_instance'
	/usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:501: in function </usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:497>
coroutine 1:
	[C]: in function 'resume'
	coroutine.wrap:21: in function <coroutine.wrap:21>
	/usr/local/share/lua/5.1/kong/init.lua:756: in function 'access'
	access_by_lua(nginx-kong.conf:87):2: in main chunk, client: ****, server: kong, request: "GET /accounts HTTP/1.1", host: "****"
10.5.4.16 - - [05/Oct/2020:14:39:16 +0000] "GET /accounts HTTP/1.1" 500 76 "-" "PostmanRuntime/7.26.5"
2020/10/05 14:39:16 [notice] 29#0: *10525438 [kong] handler.lua:?:56 "ACL" nil, client: ****, server: kong, request: "GET /accounts HTTP/1.1", host: "****"
2020/10/05 14:39:16 [notice] 29#0: *10525438 [kong] handler.lua:?:57 "Configuration collection is" "accounts", client: ****, server: kong, request: "GET /accounts HTTP/1.1", host: "****"

[YegorMedvedev]

It looks like it happens when a lot of requests come in, however, neither pod wasn't restarted nor additional ingress logs added.

The issue is also appearing with image kong:2.1.4-ubuntu

[YegorMedvedev]

Eventually, I found a bug in my plugin source code, those were fixed and the issue doesn't appear anymore. Seems to be Kong didn't kill the process because there were some open threads in my Golang plugin (e.g. opened UDP connection for remote logging system). Though, I've spent a couple of days zigging and zagging around Kong source code in order to find an issue cause.

Would be better to improve logs, or add more metrics whose will be showing stuck processes and etc.

[lampnick]

@YegorMedvedev I have the same problem with you.See this link. #6500

[YegorMedvedev]

@bungle , it's a blocker issue, please have a look...

[bungle]

@javierguerragiraldez / @gszr, does this ring any bells (I think you have a way better understanding of our Go support than I have)?

[javierguerragiraldez]

yes, working on it

[YegorMedvedev]

@javierguerragiraldez , @bungle , should I be waiting for the official release, or there is a chance to get the Docker image earlier?

[tuanpm90]

Hi @YegorMedvedev ,
can you give me your email, please?
i want to ask some things about go plugin
thanks.

[YegorMedvedev]

@tuanpm90 , fee free to use a temporary chat https://tempchat.in/c/kong-go-pl

[YegorMedvedev]

@javierguerragiraldez , @gszr, @bungle guys, unfortunately, the issue isn't fixed. After a week of running it on dev, I've got the same issue again... please find logs below:

2020/12/08 10:13:06 [error] 30#0: *39049961 [kong] go.lua:436 [jwt-b2c] starting instance: no data, client: ****, server: kong, request: "PATCH /accounts HTTP/1.1", host: "****"
2020/12/08 10:13:06 [error] 30#0: *39049961 [kong] init.lua:255 [jwt-b2c] /usr/local/share/lua/5.1/kong/db/dao/plugins/go.lua:439: no data, client: ****, server: kong, request: "PATCH /accounts HTTP/1.1", host: "****"

I'm using the kong:2.2.1-ubuntu Docker image and Kubernetes ingress.

[motiisr]

this error is happens to me as well, is there any work done on it?

[javierguerragiraldez]

for Kong 2.3 we've rewritten several parts of this code, keeping an eye on this issue and other similar ones. Please try the beta we released yesterday.

[motiisr]

thank you :) also is upgrading go-pluginserver from 0.4.0 to 0.6.0 might solve it also?

[lampnick]

thank you :) also is upgrading go-pluginserver from 0.4.0 to 0.6.0 might solve it also?

maybe, see https://github.com/Kong/go-pluginserver/pull/38