bug: APISIX3.5 has high memory usage problems #10349

Nobilta · 2023-10-17T03:56:10Z

Current Behavior

Since using APISIX3.5, the memory has been growing continuously, as shown in the figure below.

Using the slabtop command, you can see that radixtree occupies a high amount of space.

There is no such problem on APISIX2.9 or APISIX3.3.

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

Freshly install and run APISIX3.5,then use ETCD as data surface.

Environment

APISIX version (run apisix version):3.5
Operating system (run uname -a):
OpenResty / Nginx version (run openresty -V or nginx -V):openresty/1.21.4.1
etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
APISIX Dashboard version, if relevant:
Plugin runner version, for issues related to plugin runners:
LuaRocks version, for installation issues (run luarocks --version):

The text was updated successfully, but these errors were encountered:

Sn0rt · 2023-10-17T10:32:51Z

thx, you Report. pls assgin to me .

reproduce step

Successfully reproduced

design

user@ubuntu -- http request --> APISIX(3.5) on centos -- http request --> upstream.

install APISIX 3.5.0

folllow cmd run@centos

sudo yum install -y https://repos.apiseven.com/packages/centos/apache-apisix-repo-1.0-1.noarch.rpm
sudo yum-config-manager --add-repo https://repos.apiseven.com/packages/centos/apache-apisix.repo
sudo yum install apisix-base-1.21.4.1.8
sudo yum install apisix-3.5.0

install route

[root@localhost ~]# cat httpbin-anything.json
{
  "methods": [
    "GET"
  ],
  "uri": "/anything",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "192.168.31.191:38080": 1
    }
  }
}


[root@localhost ~]# curl http://127.0.0.1:9180/apisix/admin/routes/1 -H  'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d @/root/httpbin-anything.json |jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   419    0   276  100   143   4307   2231 --:--:-- --:--:-- --:--:--  4451
{
  "key": "/apisix/routes/1",
  "value": {
    "methods": [
      "GET"
    ],
    "update_time": 1697698947,
    "uri": "/anything",
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "192.168.31.191:38080": 1
      },
      "hash_on": "vars",
      "scheme": "http",
      "pass_host": "pass"
    },
    "create_time": 1697685866,
    "priority": 0,
    "status": 1,
    "id": "1"
  }
}

install benchmark tools

install https://github.com/giltene/wrk2 as benchmark tools

send request to /anythinga (not correct route)

./wrk -t2 -c100 -d10000s -R200000 --latency http://192.168.122.145:9080/anythinga

get the slab info from slabtop

11886  11886 100%    0.57K    849	 14	 6792K radix_tree_node

13692  13692 100%    0.57K    978	 14	 7824K radix_tree_node

and C+c

ubuntu@ubuntu-server:~/wrk2$ ./wrk -t2 -c100 -d10000s -R200000 --latency http://192.168.122.145:9080/anythinga
Running 167m test @ http://192.168.122.145:9080/anythinga
....
  166199.295     1.000000      5290366          inf
#[Mean    =    77988.014, StdDeviation   =    43686.962]
#[Max     =   166068.224, Total count    =      5290366]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------
  5634121 requests in 3.17m, 1.18GB read
  Non-2xx or 3xx responses: 5634121
Requests/sec:  29613.49
Transfer/sec:      6.35MB

to insepct the slab info . and found the slab not be recycle

13692  13692 100%    0.57K    978	 14	 7824K radix_tree_node

swith to correct route

ubuntu@ubuntu-server:~/wrk2$ ./wrk -t2 -c100 -d10000s -R200000 --latency http://192.168.122.145:9080/anything
Running 167m test @ http://192.168.122.145:9080/anything
  2 threads and 100 connections

inpsect

13720  13720 100%    0.57K    980	 14	 7840K radix_tree_node

stop benchmark with correct url. Found that OBJS has not become less

13748  13748 100%    0.57K    982	 14	 7856K radix_tree_node

The memory did not decrease quickly, so switch to the wrong URL for stress testing.

ubuntu@ubuntu-server:~/wrk2$ ./wrk -t2 -c100 -d10000s -R200000 --latency http://192.168.122.145:9080/anythinga
Running 167m test @ http://192.168.122.145:9080/anythinga
  2 threads and 100 connections
  Thread calibration: mean lat.: 4034.417ms, rate sampling interval: 14827ms
  Thread calibration: mean lat.: 4074.969ms, rate sampling interval: 14852ms

Memory increases very quickly

14574  14574 100%    0.57K   1041	 14	 8328K radix_tree_node

about 10 minutes later

16870  16870 100%    0.57K   1205	 14	 9640K radix_tree_node

16884  16884 100%    0.57K   1206	 14	 9648K radix_tree_node

Sn0rt · 2023-10-20T03:54:47Z

Stage information:

From the current perspective of slab leaks in Linux, it cannot be concluded that APISIX also has memory leaks.
APISIX's radixtree-based route matching is completely in user mode and has nothing to do with the kernel data structure radix_tree_node.
APISIX is implemented based on NGX. Because ngx has its own memory management, from the perspective of the operating system, the ngx memory has not been released.
User feedback The reason why 3.3 APISIX is ok is that the operating system kernel is different. 3.3 runs on the tlinux kernel.

further processing

Wait for the user to switch to tlinux and retest.

shreemaan-abhishek · 2023-10-21T03:38:25Z

@Sn0rt thanks for the detailed information.

Nobilta · 2023-10-24T02:31:54Z

Stage information:

From the current perspective of slab leaks in Linux, it cannot be concluded that APISIX also has memory leaks.

APISIX's radixtree-based route matching is completely in user mode and has nothing to do with the kernel data structure radix_tree_node.

APISIX is implemented based on NGX. Because ngx has its own memory management, from the perspective of the operating system, the ngx memory has not been released.

User feedback The reason why 3.3 APISIX is ok is that the operating system kernel is different. 3.3 runs on the tlinux kernel.

further processing

Wait for the user to switch to tlinux and retest.

After upgrading the kernel to 5.4.119-19.0009.32, the phenomenon is the same as before. The openresty worker memory keeps growing after receiving traffic. In the same environment as APISIX 3.3, using the same plug-in configuration (different data plane), version 3.5 has a memory leak problem, but version 3.3 works normally.

Nobilta · 2023-10-24T04:28:54Z

Stage information:

From the current perspective of slab leaks in Linux, it cannot be concluded that APISIX also has memory leaks.

APISIX's radixtree-based route matching is completely in user mode and has nothing to do with the kernel data structure radix_tree_node.

APISIX is implemented based on NGX. Because ngx has its own memory management, from the perspective of the operating system, the ngx memory has not been released.

User feedback The reason why 3.3 APISIX is ok is that the operating system kernel is different. 3.3 runs on the tlinux kernel.

further processing

Wait for the user to switch to tlinux and retest.

After upgrading the kernel to 5.4.119-19.0009.32, the phenomenon is the same as before. The openresty worker memory keeps growing after receiving traffic. In the same environment as APISIX 3.3, using the same plug-in configuration (different data plane), version 3.5 has a memory leak problem, but version 3.3 works normally.

According to the memory data of the last dump, the largest address space occupied after string processing is the upstream node information related to the use of the traffic-split plugin.The upstream node information that appears in the dump file is all part 1 in the figure, and the node 2 almost does not exist in the dump file. so we suspect whether it is related to the modification of the traffic-split plugin in version 3.5.

Sn0rt · 2023-10-31T10:03:53Z

Combining your information, I very much suspect that the problem was introduced in 3.4; combined with the changelog, I guess it is caused by the modification of etcd. Can you help me test it? (I tried to reproduce it on centos but failed. )

Use the 3.3 etcd file to replace the 3.5 one. Take a look at wget https://raw.githubusercontent.com/apache/apisix/release/3.3/apisix/core/config_etcd.lua

TakiJoe · 2023-10-31T10:06:33Z

Combining your information, I very much suspect that the problem was introduced in 3.4; combined with the changelog, I guess it is caused by the modification of etcd. Can you help me test it? (I tried to reproduce it on centos but failed. )

Use the 3.3 etcd file to replace the 3.5 one. Take a look at wget https://raw.githubusercontent.com/apache/apisix/release/3.3/apisix/core/config_etcd.lua

OK, I'll try to reproduce & test it

deluxor · 2023-11-03T17:37:10Z

We also spotted the same issue with the recent update of version 3.5

We only change Apisix version, same kernel version, same OS version

Kept rising until OOM

monkeyDluffy6017 · 2023-11-03T17:43:32Z

@deluxor we are tracking this problem, do you have any more clues?

TakiJoe · 2023-11-20T03:39:52Z

Here is the captured flame graphs and memory usage.

monkeyDluffy6017 · 2023-12-08T08:52:27Z

We have located this problem: #10614 and #10671, it will affect all versions between 3.4.0 and 3.7.0

monkeyDluffy6017 · 2023-12-12T06:44:19Z

I will close this issue since it has been fixed

TakiJoe · 2023-12-27T08:46:19Z

I did a grayscale test in the production environment and it looks fine so far.（apisix3.5 patched with this two fix #10614
#10671 )

zsmlinux · 2024-10-28T02:54:51Z

Here is the captured flame graphs and memory usage.

Excute me, could you tell me the steps for how to generate memory flame graph?

github-project-automation bot added this to Apache APISIX backlog Oct 17, 2023

moonming moved this to 🏗 In progress in Apache APISIX backlog Oct 17, 2023

AlinsRan assigned Sn0rt Oct 17, 2023

shreemaan-abhishek added the wait for update wait for the author's response in this issue/PR label Oct 23, 2023

github-actions bot added user responded and removed wait for update wait for the author's response in this issue/PR labels Oct 24, 2023

channer99 mentioned this issue Oct 26, 2023

help request: memory growing up #10392

Closed

Revolyssup assigned monkeyDluffy6017 Nov 13, 2023

monkeyDluffy6017 mentioned this issue Dec 7, 2023

fix: memory leak caused by timer that never quit #10614

Merged

5 tasks

monkeyDluffy6017 closed this as completed Dec 12, 2023

github-project-automation bot moved this from 🏗 In progress to ✅ Done in Apache APISIX backlog Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug: APISIX3.5 has high memory usage problems #10349

bug: APISIX3.5 has high memory usage problems #10349

Nobilta commented Oct 17, 2023 •

edited

Loading

Sn0rt commented Oct 17, 2023 •

edited

Loading

Sn0rt commented Oct 20, 2023

shreemaan-abhishek commented Oct 21, 2023

Nobilta commented Oct 24, 2023

Nobilta commented Oct 24, 2023

Sn0rt commented Oct 31, 2023

TakiJoe commented Oct 31, 2023

deluxor commented Nov 3, 2023 •

edited

Loading

monkeyDluffy6017 commented Nov 3, 2023

TakiJoe commented Nov 20, 2023

monkeyDluffy6017 commented Dec 8, 2023 •

edited

Loading

monkeyDluffy6017 commented Dec 12, 2023 •

edited

Loading

TakiJoe commented Dec 27, 2023

zsmlinux commented Oct 28, 2024

bug: APISIX3.5 has high memory usage problems #10349

bug: APISIX3.5 has high memory usage problems #10349

Comments

Nobilta commented Oct 17, 2023 • edited Loading

Current Behavior

Expected Behavior

Error Logs

Steps to Reproduce

Environment

Sn0rt commented Oct 17, 2023 • edited Loading

reproduce step

design

install APISIX 3.5.0

install route

install benchmark tools

Sn0rt commented Oct 20, 2023

shreemaan-abhishek commented Oct 21, 2023

Nobilta commented Oct 24, 2023

Nobilta commented Oct 24, 2023

Sn0rt commented Oct 31, 2023

TakiJoe commented Oct 31, 2023

deluxor commented Nov 3, 2023 • edited Loading

monkeyDluffy6017 commented Nov 3, 2023

TakiJoe commented Nov 20, 2023

monkeyDluffy6017 commented Dec 8, 2023 • edited Loading

monkeyDluffy6017 commented Dec 12, 2023 • edited Loading

TakiJoe commented Dec 27, 2023

zsmlinux commented Oct 28, 2024

Nobilta commented Oct 17, 2023 •

edited

Loading

Sn0rt commented Oct 17, 2023 •

edited

Loading

deluxor commented Nov 3, 2023 •

edited

Loading

monkeyDluffy6017 commented Dec 8, 2023 •

edited

Loading

monkeyDluffy6017 commented Dec 12, 2023 •

edited

Loading