Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

harmony invoked OOM killer #3709

Closed
LeoHChen opened this issue May 16, 2021 · 0 comments
Closed

harmony invoked OOM killer #3709

LeoHChen opened this issue May 16, 2021 · 0 comments
Assignees
Labels
bug Something isn't working high priority high priority issue with customer impact

Comments

@LeoHChen
Copy link
Contributor

Describe the bug
Validators recently noticed a higher rate of signature drop on nodes in all shards.

Using dmesg | grep harmony and you can see oom killer.

ubuntu@ip-172-31-36-171:~$ dmesg | grep harmony
[1550230.874887] harmony invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[1550230.874892] CPU: 1 PID: 42963 Comm: harmony Not tainted 5.4.0-1041-aws #43-Ubuntu
[1550230.874997] [  32209]  1000 32209  1395312   909707  7720960        0             0 harmony
[1550230.874998] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/harmony.service,task=harmony,pid=32209,uid=1000
[1550230.875042] Out of memory: Killed process 32209 (harmony) total-vm:5581248kB, anon-rss:3638828kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:7540kB oom_score_adj:0
[1550231.167912] oom_reaper: reaped process 32209 (harmony), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[2090066.261310] [  68640]  1000 68640  1400288   908756  7692288        0             0 harmony
[2090066.261313] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/harmony.service,task=harmony,pid=68640,uid=1000
[2090066.261360] Out of memory: Killed process 68640 (harmony) total-vm:5601152kB, anon-rss:3635024kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:7512kB oom_score_adj:0
[2090066.570406] oom_reaper: reaped process 68640 (harmony), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
@LeoHChen LeoHChen added the bug Something isn't working label May 16, 2021
@LeoHChen LeoHChen added the high priority high priority issue with customer impact label May 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working high priority high priority issue with customer impact
Projects
None yet
Development

No branches or pull requests

3 participants