DB Error 1040: Shlink hogging database connections resulting in too many connections error #2341
Comments
|
Thanks! If it was introduced recently, it was definitely not intentional 😅 I'll take a look. |
|
I assume you have started to experience this with v4.4.0, but it worked as expected before? |
|
I have been running Shlink for almost 2 years without any issues, but this problem seems very unusual to me and might be a side effect of something else. Shlink was automatically upgraded to version The issue first occurred yesterday, and after checking again today, the problem still persists. By the way, the logs are exposing sensitive information such as the database username, password, API keys, etc. Restarting Shlink closes all connections and using it for a couple minutes there are no persistent connections open. So the problem seems to develop over time. Looking at the db process list timestamps, many connections appear to open at the exact same second. Does Shlink perhaps use a worker queue, cron job, or timer that might trigger this behavior? Since I limited Shlink to 25 connections, it is now constantly logging errors. Perhaps these logs provide more clues: Logs <---- [SPOILER - Click to Open] |
|
Ok, I think I know what's very likely the root cause. Shlink does in fact run background asynchronous jobs, and it has a mechanism to close the connections opened there to avoid issues like this one. This logic is set on a per job basis, so we don't run logic designed to open/close database connections for jobs which do not interact with the database at all. The problem is that Shlink 4.4.0 changed the logic in a job that didn't use to interact with the database, but now does, and I forgot to enable the open/close connection logic for it. Fixing it should be easy. |
|
I have just released Shlink 4.4.1, which includes a fix for this. The docker image will be available in ~20 minutes, once it finishes building. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Shlink version
docker-stable-4.4.0
PHP version
docker-stable-8.4.2
How do you serve Shlink
Docker image
Database engine
MariaDB
Database version
11.4.4-MariaDB
Current behavior
shlink is opening a lot of db connections and never closing them resulting in all db users being denied due to
too many connectionsThis killed all other services i'm running and i had to limit connections by user to prevent this happening again.
Is this some bug that was introduced recently ? https://hub.docker.com/layers/shlinkio/shlink/stable/images/sha256-15283c2c7924baf289688f059c1ef9ec907c7f06a5d9629c36f3801734955c21
Expected behavior
close or reuse db connections
sure i can increase connection limits but there seems to be no end, so that's not going to fix it.
Minimum steps to reproduce
Running the official shlink server, stable tag, nothing special.
I limited connections to 25 per user and shlink got that in under 6 hours again, killing shlink.
100 is the default mariadb limit, afterwards everyone will be locked out and only shlink has that many connections.
The text was updated successfully, but these errors were encountered: