High CPU usage on docker

[rayures]

Version

latest docker
chrisleekr/binance-trading-bot:latest@sha256:d79e5410e42884eee835f652fb78c8c7c5bbe814bad3967057cd6d5f5ed96ce5

Running Version: v0.0.74 (67a4adf)
Latest Version:
v0.0.74

Description

the binance trade bot docker container is continuously using 30-40 % cpu time. (top in host, and docker stats on host). Host is a 4 core i5 machine. No limits on the container.

host:
i5 4c
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-17-amd64
Architecture: x86-64
docker:
Docker version 19.03.5, build 633a0ea838

docker stats:

in the binance bot container:

is this normal behavior?

[habibalkhabbaz]

I am having high RAM usage with Docker after running the bot for couple of hours.
However, I don't think the issue is related to the bot but it is related to the Docker instead as many people complaining about high usage problem using Docker on Windows.
I tried to apply limits using WSL in Windows but no luck, still it is behaving strange when it's reached the maximum allowed value and my PC being heavy.
I am trying to find a solution for this 🙅‍♂️

[pedrohusky]

I am having high RAM usage with Docker after running the bot for couple of hours.
However, I don't think the issue is related to the bot but it is related to the Docker instead as many people complaining about high usage problem using Docker.
I tried to apply limits using WSL in Windows but no luck, still it is behaving strange when it's reached the maximum allowed value and my PC being heavy.
I am still trying to find a solution for this 🙅‍♂️

Maybe there is a way for us to not use docker?

Like, start the bot right from terminal?

[chrisleekr]

@rayures

The bot running in RPi4 uses 30~40% CPU as well, which makes sense since it's so low spec.

But the bot running in i5/8GB Ubuntu server(18 Bionic Feaver) running 4~10% CPU only.

It seems you are using Ubuntu 10 (Buster), which is very old. Is there any specific reason that you are using that version?

Like, start the bot right from terminal?

If you install redis/mongo separately, it is definitely possible, but I don't think that will benefit anyway because Docker itself uses native resources.

[habibalkhabbaz]

I am having high RAM usage with Docker after running the bot for couple of hours.
However, I don't think the issue is related to the bot but it is related to the Docker instead as many people complaining about high usage problem using Docker.
I tried to apply limits using WSL in Windows but no luck, still it is behaving strange when it's reached the maximum allowed value and my PC being heavy.
I am still trying to find a solution for this 🙅‍♂️

Maybe there is a way for us to not use docker?

Like, start the bot right from terminal?

Yes, there is a way for this but the problem is we have to install everything separately, such as mongodb and redis on our local OS. That's why I prefer Docker.
I am thinking of installing Ubuntu on my PC but no space xD

[pedrohusky]

Like, start the bot right from terminal?

If you install redis/mongo separately, it is definitely possible, but I don't think that will benefit anyway because Docker itself uses native resources.

Hm. Can you explain me how to do it? I would like to test it without the docker, as my machine has low ram (8gb), and docker eats it all, even with the wslconfig thing done.

I think this will fix many problems, like:

• Clock desync, that is a constant docker problem.
• Memory/CPU will not be abused.
• Even the changes in the project will be better with the terminal, as it is faster than docker in any way.

But, as you said, docker manages it all, and it is compatible to multiple systems.

I think it will benefit us.

[chrisleekr]

@pedrohusky
I do not have Windows machine so I cannot tell how exactly.

Briefly, install Redis, Mongo.
Simply point them by updating config.

• https://github.com/chrisleekr/binance-trading-bot/blob/master/config/custom-environment-variables.json#L14

[rayures]

@rayures

The bot running in RPi4 uses 30~40% CPU as well, which makes sense since it's so low spec.

But the bot running in i5/8GB Ubuntu server(18 Bionic Feaver) running 4~10% CPU only.

It seems you are using Ubuntu 10 (Buster), which is very old. Is there any specific reason that you are using that version?

cat /etc/debian_version
10.10

its Debian at its latest version.... Why do you think its old?

[chrisleekr]

Oh sorry, I thought it's Ubuntu.... Ubuntu Buster was v10.

I am not sure why it is eating CPU up that much despite you also have i5/8G RAM.

[rayures]

upgraded Debian 10.10 to 11.0
upgraded Docker 19.03.5 to 20.10.7

still high cpu load (about 40%).

what changed between 0.0.65 and 0.0.66?
reverted to 0.0.65 and it has about 2% cpu usage.
upgraded tot 0.0.66 and it goes to 40% continuously.

@chrisleekr

changelog states:

Updated frontend to display version - #59
Support monitoring multiple coins simultaneously - #77
Added docker-stack.yml for the Portainer - @hipposen
Fixed precision issues for some FIAT - #90
Improved frontend & settings UI - #93 #85
Support all symbols - #104
Added stop loss feature - #99
Stabilised Local Tunnel, cronjob and Binance WebSocket
Prevented to place new order when the API limit reached - #163
Added NPM task for building docker image on Windows by @garyng - #175

[vuongductuanktmt]

Same issue please fix it. Thanks sir

---

[chrisleekr]

@rayures Sorry for the late reply.
The main change in that release is Support monitoring multiple coins simultaneously - #77, which actively checks the multiple symbols.

@vuongductuanktmt
I don't think I can fix it. There must be something else is going on.

My latest version in RPi is not even using 50%.

[bangbaew]

@rayures Sorry for the late reply.
The main change in that release is Support monitoring multiple coins simultaneously - #77, which actively checks the multiple symbols.

@vuongductuanktmt
I don't think I can fix it. There must be something else is going on.

My latest version in RPi is not even using 50%.

I'm also using RPi 4 with latest bot version (0.0.74), same OS as yours

but the container constantly exits and restarts itself, looks like the CPU is being excessively used

From my Grafana monitor, you can see that the temperature is constantly getting up (around 40c), this is when the container is running, and down (around 35c), this is when the container is exiting and trying to restart, the Grafana is also slow to visit, I'm not sure if it's Docker problem, what should I do to fix it?

looks like it's using around 20% cpu for node in top command but excessive cpu usage in docker stats

[chrisleekr]

Hey @bangbaew

Can you try with disabling swap?

1. sudo nano /etc/dphys-swapfile

2. change CONF_SWAPSIZE=0

3. Save and Reboot

[bangbaew]

Hey @bangbaew

Can you try with disabling swap?

1. sudo nano /etc/dphys-swapfile
2. change CONF_SWAPSIZE=0
3. Save and Reboot

Ok, I'll try it and give you an update,
but in my comment I attached the wrong docker stats picture.
I was meant to attach this picture

[chrisleekr]

Compare with mine.

Disable swap may help to release block I/O.

[bangbaew]

Compare with mine.

Disable swap may help to release block I/O.

Looks like the bot is still exiting and restarting...

even I've disabled swapfile and block i/o is 0 bytes

My RPi was really slow before the container exited, looked like it made my RPi temporarily down...

what build stage are you using? I'm currently using the full production build stage.

This is my compose file, I saw it was using development build stage with binance test mode when I downloaded it,
so I changed to production stage with live mode.
docker-compose.yml.zip

[chrisleekr]

Hmmm, I don't know why are you running docker-compose.yml.
As I instructed in the README.md, use this method.

Or if using Raspberry Pi 4 32bit, must build again for Raspberry Pi:

npm run docker:build
docker-compose -f docker-compose.rpi.yml up -d

FYI, I found it looks stuck at building step, but it is not, takes really long time to build.

What I am trying to do is to distribute docker image for ARM/v7, which is RPi, then you won't need to build the image. Just need docker image.

[chrisleekr]

And for the restarting, have you looked at the log?

https://github.com/chrisleekr/binance-trading-bot/wiki/Troubleshooting#how-to-get-the-logs-by-installing-bunyan-npm-package

[bangbaew]

Hmmm, I don't know why are you running docker-compose.yml.
As I instructed in the README.md, use this method.

Or if using Raspberry Pi 4 32bit, must build again for Raspberry Pi:

npm run docker:build
docker-compose -f docker-compose.rpi.yml up -d

FYI, I found it looks stuck at building step, but it is not, takes really long time to build.

What I am trying to do is to distribute docker image for ARM/v7, which is RPi, then you won't need to build the image. Just need docker image.

Because I used docker-compose.rpi.yml and it pulled your image from the docker hub which was build for x86 and it ended up with standard_init_linux.go:219: exec user process caused: exec format error like this thread
perara/wg-manager#69

but I didn't understand the npm run docker:build command so I didn't run it, I thought it was like normal Dockerfile building, so I used build: . instead, it didn't stuck but it just took a quite long time to build.

I'll try to use npm run docker:build

[bangbaew]

I'll try to use npm run docker:build

I've tried npm run docker:build, I think it's the same as normal Dockerfile building as I usually do, the container was still crashing, I think the problem is with monitoring symbols every second, which consumes a lot of cpu, because I had more than 30 symbols monitoring simultaneously and the timer took a long time to refresh before crashing, so I cleaned the volumes to default 4 symbols monitored, the bot works smoothly, the timer increases second by second without crashing.

Can you add the option to adjust monitoring interval? I think it could be monitored every minute like the bot before monitoring-coins-simultaneously release. my RPi worked great with that version.

[chrisleekr]

I really don't think it's the issue with the bot because I use it without much hassle.

What SD card are you using? Are you using 10 Speed Class?

(Updated: image for sd card was not micro sd card.)

Crashing is not making sense as well. Have you checked the log? What did the log say when it's crashed?

In anyway, if you want to increase interval, refer these environments parameters you can set.

https://github.com/chrisleekr/binance-trading-bot/blob/master/config/custom-environment-variables.json#L86
https://github.com/chrisleekr/binance-trading-bot/blob/master/config/custom-environment-variables.json#L213

[bangbaew]

I really don't think it's the issue with the bot because I use it without much hassle.

What SD card are you using? Are you using 10 Speed Class?

(Updated: image for sd card was not micro sd card.)

Crashing is not making sense as well. Have you checked the log? What did the log say when it's crashed?

In anyway, if you want to increase interval, refer these environments parameters you can set.

https://github.com/chrisleekr/binance-trading-bot/blob/master/config/custom-environment-variables.json#L86
https://github.com/chrisleekr/binance-trading-bot/blob/master/config/custom-environment-variables.json#L213

The log didn't say error messages, they were just normal bot messages log but it was so slow logging and eventually crashed so I didn't capture any screenshot of it, my SD card is SanDisk class 10 UHS-I

thanks for suggesting me the environment variable, may I ask what's the unit of it? seconds?

[chrisleekr]

The variable values can be referred in this file - https://github.com/chrisleekr/binance-trading-bot/blob/master/config/default.json#L50

It's crontab with milliseconds.

[rayures]

thnx for the reply.

as a test i increased the delay in the default config ( crontime ) to double the amount. Can see it directly in cpu usage. Switching between 2% and 40%. So it definitely is the coin updates.

[bangbaew]

The variable values can be referred in this file - https://github.com/chrisleekr/binance-trading-bot/blob/master/config/default.json#L50

It's crontab with milliseconds.

I've set BINANCE_JOBS_TRAILING_TRADE_CRON_TIME=*/10 * * * * *
it refreshes every 10 seconds and it now consumes much lower cpu usage!,

from 10:46 to 11:00 was when I set crontime to 10 seconds (< 25% cpu usage), the rest were every second (<= 75%).
my RPi also works fine within 5 seconds crontime (<= 50% cpu usage).

but I don't quite understand what BINANCE_JOBS_TRAILING_TRADE_INDICATOR_CRON_TIME does,
I didn't see the effects of configuring it, can you please explain?
Thanks.

[chrisleekr]

@bangbaew

As I said, this cronjob is supporting seconds as well. So it's 6 *, which means every second.
* * * * * *

If you set */10 * * * * *, it will run every 10 seconds.

BINANCE_JOBS_TRAILING_TRADE_INDICATOR_CRON_TIME is another cronjob which requires for operating the bot.
See this file to understand how it works - https://github.com/chrisleekr/binance-trading-bot/blob/master/app/cronjob/trailingTradeIndicator/steps.js

[vuongductuanktmt]

bot doesn't work like the time frame on the photo below, I checked the current prices of the coins and didn't see any updates. I had to restart and the bot worked fine.
I had to use crontab every 8 hours to restart the bot to solve this problem, do you have a better solution.

Thanks sir

[bangbaew]

bot doesn't work like the time frame on the photo below, I checked the current prices of the coins and didn't see any updates. I had to restart and the bot worked fine.
I had to use crontab every 8 hours to restart the bot to solve this problem, do you have a better solution.

Thanks sir

Do you have 30++ symbols monitored simultaneously every second? It's really consuming the CPU.
I also had the same problem, so I have to increase the monitoring time to every 10 seconds and now it works flawlessly.

[chrisleekr]

@vuongductuanktmt

I had to use crontab every 8 hours to restart the bot to solve this problem, do you have a better solution.

I also restart the bot every 3 hours in Raspberry Pi, but it's not because of the performance. I do that because of the local tunnel.
I will check the code and try to improve the performance, but not sure whether it's possible. 🤷‍♂️

[chrisleekr]

Hey,

Can you check the redis log whether you see the below message?

$ docker logs binance-redis -f

...
2956:C 23 Aug 2021 11:29:03.109 * RDB: 0 MB of memory used by copy-on-write
1:M 23 Aug 2021 11:29:03.164 * Background saving terminated with success
1:M 23 Aug 2021 11:29:13.009 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 23 Aug 2021 11:29:15.011 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 23 Aug 2021 11:30:04.067 * 10000 changes in 60 seconds. Saving...
...

[rayures]

currently i am running pedrohusky fork, that one runs about 1% cpu (all containers combined). Will try to switch back and look at the redis log.

[vuongductuanktmt]

Hey,

Can you check the redis log whether you see the below message?

$ docker logs binance-redis -f

...
2956:C 23 Aug 2021 11:29:03.109 * RDB: 0 MB of memory used by copy-on-write
1:M 23 Aug 2021 11:29:03.164 * Background saving terminated with success
1:M 23 Aug 2021 11:29:13.009 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 23 Aug 2021 11:29:15.011 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 23 Aug 2021 11:30:04.067 * 10000 changes in 60 seconds. Saving...
...

4189:C 19 Aug 2021 23:17:08.142 * RDB: 0 MB of memory used by copy-on-write
1:M 19 Aug 2021 23:17:08.182 * Background saving terminated with success
1:M 19 Aug 2021 23:18:08.000 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 19 Aug 2021 23:18:09.019 * 10000 changes in 60 seconds. Saving...

[chrisleekr]

What I found is, Redis is trying to save the changes to the disk so it doesn't lose the data. It's called AOF.

1:M 19 Aug 2021 23:18:08.000 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

Since Raspberry Pi is not using SSD or a faster disk, it causes a bottleneck.

A Redis is just a temporary cache, it should be able to flush anytime.
However, some features in the bot are relying on the Redis to be persistent.

I am testing the code with the following changes:

- Added indexes for MongoDB
- Updated configuration to use cache as well to improve performance
- Updated redis.conf to disable AOF for improving performance
- Moved grid/manual orders to MongoDB from Redis since Redis is not persistent anymore

Although it's not improving performance in the Raspberry Pi significantly since it still requires read/write to MongoDB for some features. However, at least Redis would not write to the disk anymore as it shouldn't be.

[chrisleekr]

I assume this is no longer the issue. I will close the issue.