502 - Bad gateway on Vesta CP admin

[treatys]

Operating System (OS/VERSION):

Debian 9.4 (x86_64)

VestaCP Version:

0.9.8 (amd64), release 21

Installed Software (what you got with the installer):

Complete production server installation including Bind, Nginx, Apache, MySQL, PgSQL and dovecot+exim.

Steps to Reproduce:

Since yesterday (2018/06/22), Vesta CP panel crashes repeatedly. It reports error 502 - Bad Gateway in nginx. In /var/log/vesta/* there are no errors about this crash reported, only in /var/log/vesta/nginx-error.log, there is mentioned:

2018/06/22 14:35:29 [error] 19059#0: *49 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: 109.238.216.159, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "taira.srv.getmedia.cz:8083"

While I restart vesta service with systemctl restart vesta, I am able to access the panel for a while, however, this is not stable and wont last long.

I don't have any idea while is this happening to me. This bug occurs on all my Vesta CP installations, so whole my cluster (3 nodes) is particullary down.

Do you have any idea what's gone wrong out there?

[treatys]

I've just bumped into this thread on forum https://forum.vestacp.com/viewtopic.php?t=17153

The problem is also mentioned on these:

• https://forum.vestacp.com/viewtopic.php?f=16&t=17157
• https://forum.vestacp.com/viewtopic.php?f=16&t=16601
• https://forum.vestacp.com/viewtopic.php?f=10&t=17159
• https://forum.vestacp.com/viewtopic.php?f=10&t=17141

That is exactly my case.

[saurabhsharma]

Hello, Is there any update on this? my server also having exact same issue.

[mehargags]

I Can confirm this happens with me too..but only on Apache-less servers that have Nginx + PHP-FPM.

If I restart vesta service, it goes good but then the next day its crashed again. I guess may some cron job is doing this ?

I tried all default Cron Jobs for user admin... all went fine but after I ran
sudo /usr/local/vesta/bin/v-update-user-stats
Vesta Panel threw 502.
I restarted service and tried it again... this time it did not crash. So can't really say it is because of some crons. I have separate cron logs enabled but can't see any error recorded.

/var/log/vesta/nginx-error.log gives
[error] 95978#0: *25 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: _, request: "GET /list/user/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "IP:8908", referrer: "https://IP:8908/login/"

I am running it on port 8908 and not 8083

[korsar4eg]

Same issue, any updates ?
P.S. Start happening couple days ago

[saurabhsharma]

I removed nginx to fix the issue, but I am thinking of moving to Virtualmin

…

________________________________ From: korsar4eg <[email protected]> Sent: Sunday, June 24, 2018 8:48 AM To: serghey-rodin/vesta Cc: Saurabh Sharma; Comment Subject: Re: [serghey-rodin/vesta] 502 - Bad gateway on Vesta CP admin (#1625) Same issue, any updates ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1625 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAv8YaOBGMGPtncbFynO2UwkkDAdbA3Dks5t_wUBgaJpZM4UzyCQ>.

[gd-euss]

Same problem

[dpeca]

What is a sys load of your servers?

[gd-euss]

sys load is low. I have got 1 website running on whole virtual server.
Vesta fails even after update.
I think problem is somewhere in the cron jobs, because it fails every x minutes after vesta service restart.
Error from nginx-error.log: 2018/06/24 17:45:22 [error] 29355#0: *29 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: (my ip), server: _, request: "GET /list/updates/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "mydomain.tld:8083", referrer: "https://mydomain.tld:8083/login/"

UPD: when appears 502 BAD Gateway, Vesta service status shows that vesta still active and running fine.

[petarkozic]

Same problem !
Same logs!
Same as gd-euss

[ram108]

My Vesta works as usual. Hetzner cloud, Ubuntu 16.04, nginx + php7.0-fpm

[treatys]

I can confirm that the problem is caused by this CRON script:

/usr/local/vesta/bin/v-update-user-stats

Everything works until the cronjob is called. Also, as mentioned before, all my servers have really low sys load.

[maciejkrol]

I have the same issue here. Ubuntu 16 AWS (t2.small)
AMI ID: ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180522 (ami-963cecf4)

@treatys @mehargags
Suspending /usr/local/vesta/bin/v-update-user-stats did not help at all.
I suspended later all CRON jobs did not help at all either.

[Skamasle]

Changing pm from dynamic to ondemand and reboot help me

[Anton98567]

@Skamasle sorry, but what "PM" means?

[anton-reutov]

@Anton98567, Process manager

[uskaritel]

Confirmed. I have the same issue on CentOS 7

[hivanski]

Freshly updated VestaCP have the same bug.

[2018-06-24] Security fix release 0.9.8-22
Debian 9

[mohamad4j]

I have same problem. Debian 9 vesta 0.9.8

[stanma9107]

I have the same problem
System:ubuntu 16.04

[Anton98567]

Same problem
Ubuntu 16.04.4 LTS
vesta 0.9.8-22

Guys, how to fix it?

tail -f /usr/local/vesta/log/nginx-error.log
 [error] 1855#0: *44 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: _, request: "POST /api/index.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host:

[serghey-rodin]

Thanks for reporting about this issue. The fix will be available soon.

[progcode]

Same as on Ubuntu 16.04

[quasiDigi]

Some error on centOs 7

tail -f /usr/local/vesta/log/nginx-error.log
     [error] 816#0: *580 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: , request: "POST /softaculous/index.php?act=cp_api&api=serialize HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: ""

[serghey-rodin]

It seems like this issue is caused by default pm.max_requests value.
Please try out the following fix

sudo wget http://c.vestacp.com/tmp/php-fpm.conf -O /usr/local/vesta/php/etc/php-fpm.conf
sudo service vesta restart

I'm going to test it next few hours and push as update if everything is fine.
Thanks

[quasiDigi]

Great. Everything seems to work fine again.

Thx @serghey-rodin

[petarkozic]

@serghey-rodin
Yes, this definitely works !

[Anton98567]

@serghey-rodin good job!

[ninjao]

@mahdiyari ok lets hope that works out for you! :)

Did you restart only the "vestacp" related services? Or did you literally restart every service on your machine? Would be good to know for future people unable to do a hard reboot.

Good luck

[mahdiyari]

502 again!
I restarted vesta related services!

[dpeca]

and deleted unix socket before restart?

[mahdiyari]

Of course!

[gd-euss]

@mahdiyari Have you tried to have a look on what services have been upgraded and restart them too?

[dpeca]

ok, let's then use safe variant that should work for sure
edit /usr/local/vesta/php/etc/php-fpm.conf
find pm=dynamic
set to pm=ondemand

and then
service vesta restart

[mahdiyari]

@gd-euss how can I find upgraded services?
@dpeca Edited and restarted.

[gd-euss]

@mahdiyari first try dpecas variant and if doesn't work for you then try to find upgrade log in your system logs.

[dpeca]

i believe it will work because that way issue will be avoided (whatever it is)

[mahdiyari]

No, again 502 error!

[anton-reutov]

@mahdiyari you should try to reboot your server

[dpeca]

does it works even one second?

[mahdiyari]

I can't reboot my server @anton-reutov
I can access admin panel after restarting vesta service, but it shows 502 after a while

[gd-euss]

@mahdiyari your logs might be in /var/log/dist-upgrade (Not sure because never used Ubuntu)

[mahdiyari]

I restarted all services (except ssh), let's see what will happen!

[ninjao]

@mahdiyari you are running out of options :/

I can think of checking all your logs, making sure to resolve everything else... and restarting other services on your machine. I am only thinking of last resort options... restart your network interface? But this might also cause issues with your blockchain node. Also be careful when restarting network interface while using SSH...

I tried so many things to get my server running again, but in the end a reboot was probably what solved the issue.

Edit: Ah I see you have done something, good luck!

[mahdiyari]

Now, I don't have any option and still, there is 502 error
A bug comes with a silent update (release 21) and we can't solve that without a reboot!
if anyone found a working solution (without reboot) please comment here
Thanks for your time:)

Update: No matter what you have in your server! You should reboot to solve. A bad development:)

[quasiDigi]

everything seems to work fine after update, upgrade and reboot.
Thx

[niccolomineo]

@mahdiyari did you solve it? I am still fighting with it.

[Skamasle]

@niccolomineo you update and reboot ?

If you not reboot server may you cant solve it

[niccolomineo]

Thank you, I restarted NGINX, I hope that sorts it!

edit: Nope, problem's still there.

I did:

rm -f /var/run/vesta-php.sock
service vesta restart
service nginx restart

[Skamasle]

Try reboot your server.

[niccolomineo]

I did and it is working now. Let's see for how long it lasts (hopefully forever).

[gd-euss]

apt-get update, apt-get upgrade, reboot will help you.

[TarKHaoS]

apt-get update, apt-get upgrade, reboot will help you.

This definitely worked for me. Cheers!

[Olegu13]

This worked:

rm -f /var/run/vesta-php.sock
service vesta restart 
service nginx restart 
reboot

[koolamusic]

Nginx just seems to have a problem in CentOS builds
It still cannot connect

[evagabond]

This worked:

rm -f /var/run/vesta-php.sock
service vesta restart 
service nginx restart 
reboot

This worked for me too!

[luckydevilru]

I can't restart service
nginx-error.log :

2020/06/02 10:18:19 [crit] 647#0: *10 connect() to unix:/var/run/vesta-php.sock failed (2: No such file or directory) while connecting to upstream, client: 176.59.34.108, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "server.domain.ru:8083"