kibana-plugin install hangs forever

[rossengeorgiev]

On CentOS 7.5 with a fresh Kibana 6.5.0 install, when I run:

/usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.7.0_6.5.0.zip

The command never finished. I found #664, which advises to Ctrl+C after there is no longer any CPU usage. That is not an acceptable solution when the plugin installation in part of automated scripts.

It apears that the issue is not isolated to this plugin alone. Please keep this open until the issue is resolved.

elastic/kibana#19678

[JuanjiJG]

Hello @rossengeorgiev,

We've been struggling with this error for a long time, and we're still working on finding a proper solution until the Elastic team fixes this. As far as we know, they're working on several improvements for the plugin optimization process.

Before proceeding, uninstall the previous Wazuh app installation by running these commands:

systemctl stop kibana
/usr/share/kibana/bin/kibana-plugin remove wazuh
rm -rf /usr/share/kibana/optimize/bundles

I suggest you installing the app with the following command:

sudo -u kibana NODE_OPTIONS="--max-old-space-size=3072" /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.7.0_6.5.0.zip

And now, restart Kibana:

systemctl restart kibana

And see if your problem gets solved. Let us know about the results.

Regards,
Juanjo

[rossengeorgiev]

Hi @JuanjiJG, Thanks for the info, I already found that in the previous issue, and that is what I ran. It still hangs at the end. My workaround for the moment is to introduce a timeout and kill it after a while.

[JuanjiJG]

Hi again @rossengeorgiev,

I'm sorry you're still facing this issue. Certainly, it's been a recurring problem for a long time. We'll let you know if we have more information or possible workarounds for this.

As you pointed out, this issue is not totally related to the Wazuh app, as many other users and developers are also having the same issue with other Kibana plugins. In any case, this is not an excuse, so we're constantly researching and working on possible solutions.

Assuming that you have all the Elastic Stack on the same machine, another suggestion is that you stop the Elasticsearch and Logstash services and try to install the app with only the Kibana service running. Maybe you already have a great amount of RAM on your system, but in some cases, we managed to install the app with this workaround.

You can also try with the --max-old-space-size flag and try different amounts of RAM, for example 8192 for 8GB of RAM.

We'll appreciate all your feedback and findings on this matter. Once more, apologies for this inconvenience.

Regards,
Juanjo

[jesusgn90]

Hi @rossengeorgiev and @JuanjiJG ,

From the Elastic forums, we know that some OS don't exit once finished, just look for the "App ready" log and top command to verify if the plugin was installed, then press ctrl + c.

Regards

[rossengeorgiev]

Hi @jesusgn90 could you elaborate? "App ready" and use of top ?

[jesusgn90]

Sure @rossengeorgiev ,

1. While Kibana is installing a plugin (any plugin) you should see a node process eating 100% of CPU and a huge amount of RAM.

2. Once it seems to be finished (node process reduces CPU/RAM usage), and if it didn't exit properly, press ctrl + c

3. Restart Kibana and look for the "App ready" log using this command:

$ cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp.log | grep -i "app ready"
{"date":"2018-11-26T09:56:21.460Z","level":"info","location":"[initialize][checkKnownFields]","message":"App ready to be used."}

I hope it helps!

Regards

[jesusgn90]

@rossengeorgiev assuming this is solved for you, re-open this ticket or open a new ticket if you are still facing troubles.

Regards!

[rossengeorgiev]

Hello, this is not solved. The issue still exist and this should remain open until the issue is resolved without workarounds.

[jesusgn90]

The problem is that is not a Wazuh related problem, it's a common issue for all the Kibana plugins... I think you should track that repository instead, for instance, the issue you pasted here: elastic/kibana#19678

Also, I can update this ticket (even if it's closed) once the Elastic team gives some light about this.

Regards