bug: dubbo-proxy timeout #10620
Comments
|
please use english to communicate and provide detailed reproduction steps. |
|
my nginx config |
|
|
my route {
"uri": "/xxxxxx",
"name": "获取组件列表",
"plugins": {
"dubbo-proxy": {
"method": "invoke",
"service_name": "xxxx",
"service_version": "1.0.0"
},
"prometheus": {},
"proxy-rewrite": {
"headers": {
"dubbo-method": "xxxx",
"dubbo-service": "xxxxx",
"parameter-types-desc": "xxxxxxx"
}
}
},
"upstream_id": "490548123844739786",
"status": 1
} |
|
my upstream {
"type": "roundrobin",
"discovery_type": "nacos",
"discovery_args": {
"namespace_id": "gateway-dev"
},
"name": "xxxx",
"service_name": "xxx"
} |
|
@shreemaan-abhishek This doesn't seem to be caused by a timeout. Because it didn't reach the default 60s timeout, and it didn't even reach 1s, but it still timed out. |
|
I test this with https://github.com/alibaba/tengine/tree/master/modules/mod_dubbo |
|
HI @ShenFeng312 , please checkout : #10765 and #10765 (comment) Does this work for you ? |
It did not work for me. When the request payload is very short, the request is successful in such cases. Currently, it seems that the issue only arises when the payload is too long. Additionally, if I use Tengine directly, it can return success. @sheharyaar |
|
Where are you attaching the payload ? In the body or in the headers ? Currently, only payload via headers is supported. |
This is unrelated to how the payload is set. You can look at my captured data. If there's an issue with the payload, it should be a problem with the Dubbo protocol. However, this is highly unlikely. Or should I declare the HTTP response content-length in the data returned by Dubbo? |
|
I will take a look at the captured packet soon. |
|
I have the same trouble , I add "dubbo_buffer_size 8096k;" in config.yaml like this:
'8096k' is enough for the dubbo respose body but apisix still return the error: |
|
@johndudo maybe u can use |
|
@ShenFeng312 http-dubbo plugin? where 's the plugin :)? or you mean to write a plugin 'http-dubbo' to proxy dubbo request? |
|
@sheharyaar @shreemaan-abhishek @johndudo Hello, I have found more information. Please assist me in reviewing it together. When reading data from upstream, APISIX invokes In n = c->recv(c, b->last, b->end - b->last);
if (n == NGX_AGAIN) {
ngx_add_timer(pc->read, u->conf->read_timeout);
}The purpose of this code is to handle read timeouts. However, I cannot understand why |
|
could be due to the openresty version upgrade? |
|
@shreemaan-abhishek The bug is caused by differences between OpenResty and Nginx. In Nginx's if (!u->request_body_sent) {
u->request_body_sent = 1;
if (u->header_sent) {
return;
}
ngx_add_timer(c->read, u->conf->read_timeout);
if (c->read->ready) {
ngx_http_upstream_process_header(r, u);
return;
}
}The added However, in OpenResty, due to the But the |
|
so how we fix it?? |
|
@ShenFeng312 at the end, what's your solution? I saw there is openresty-12.5.3.2 patch for ngx_multi_upstream_module. But we still have the same issues.
|
|
@qiyuan4f see this api7/ngx_multi_upstream_module#16 |
|
@ShenFeng312 thanks, the implementation is poor, but stable than the lua version. http-dubbo plugin is much worse than this one. I am trying to rewrite an new version based on the fastjson2, the serialization part will take time to verify. Any other idea? |
|
So far my headcache is OOM, the APISIX 3.10.0 with dubbo-proxy easily to make system OOM. try to analysis deeply, but it seem that also impact by other plugins. |
|
@ShenFeng312 applied the patch, but I saw the wrong request error were increased
|
|
do you have wechat number? can you send your wechat number to my email [email protected]. we can talk easy ! sorry for my poor english @qiyuan4f |
@qiyuan4f Rewrite new C++ plugin is a good idea,Previously, I was considering implementing a new C++ plugin to facilitate the use of ngx_multi_upstream_module. However, this module was developed for NGINX, and in fact, it is incompatible with OpenResty. If we were to develop a new C++ plugin, we would need to understand the implementation of both ngx_multi_upstream_module and dubbo-proxy, which would take a considerable amount of time. I apologize for not paying much attention to the open-source community recently due to a busy work schedule. As far as I know, Fastjson2 is based on JSONB, which may not be very suitable for gateway. If possible, I believe the older Fastjson serialization protocol is more appropriate for gateway use. |
|
@ShenFeng312 send you email, my email is: [email protected] |
Current Behavior
2023/12/08 08:10:07 [warn] 45#45: *57631 multi: multi connection detach not empty 00007F45BD856FA0
2023/12/08 08:10:07 [error] 45#45: *57627 upstream timed out (110: Operation timed out) while connecting to upstream, client: 39.99.246.117, server: _, subrequest: "/app/Service/getxxxx", upstream: "dubbo://39.101.204.9:20918"
39.99.246.117 - - [08/Dec/2023:08:10:07 +0000] 39.99.253.87:30004 "POST /app/Service/getxxxx HTTP/1.1" 504 164 0.026 "-" "PostmanRuntime/7.35.0" 39.101.204.9:20918 504 0.026 "http://39.99.253.87:30004/app/Service/getxxxx"
Expected Behavior
response success
Error Logs
2023/12/08 08:10:07 [warn] 45#45: *57631 multi: multi connection detach not empty 00007F45BD856FA0
2023/12/08 08:10:07 [error] 45#45: *57627 upstream timed out (110: Operation timed out) while connecting to upstream, client: 39.99.246.117, server: _, subrequest: "/app/Service/getxxxx", upstream: "dubbo://39.101.204.9:20918"
39.99.246.117 - - [08/Dec/2023:08:10:07 +0000] 39.99.253.87:30004 "POST /app/Service/getxxxx HTTP/1.1" 504 164 0.026 "-" "PostmanRuntime/7.35.0" 39.101.204.9:20918 504 0.026 "http://39.99.253.87:30004/app/Service/getxxxx"
tcpdump info
Steps to Reproduce
Environment
The text was updated successfully, but these errors were encountered: