Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container ot-es is Unhealthy #6

Open
shaileshchaudhary11 opened this issue Sep 27, 2024 · 15 comments
Open

Container ot-es is Unhealthy #6

shaileshchaudhary11 opened this issue Sep 27, 2024 · 15 comments
Assignees

Comments

@shaileshchaudhary11
Copy link

shaileshchaudhary11 commented Sep 27, 2024

Hi,

I am encountering an issue with the ot-es container in my setup. When I attempt to run the application, the container is marked as unhealthy, and I receive the following error:

image

make: *** [Makefile:123: platform_up] Error 1

I've tried restarting the container and checking the logs for any clues, but I am unable to resolve the issue. I am attaching
ot-es.log
log file for further details below.

Here are the details of my setup:

Docker version 27.3.1, build ce12230
Operating System: Ubuntu 22.04.5 LTS
Profile: config.2406-gcs
OpenJDK version 11.0.24

I am following blog ( https://blog.opentargets.org/standalone-deployment-tool/ )
for setting up open target.

Could any one please assist me in troubleshooting this issue? Any help would be greatly appreciated!

Thank you!

@mbdebian
Copy link
Collaborator

Thanks for reporting @shaileshchaudhary11 , would you mind providing the configuration profile you're using, so we can reproduce the issue on our side and investigate what may be happening?
Thanks!

@mbdebian mbdebian self-assigned this Sep 27, 2024
@remo87
Copy link
Contributor

remo87 commented Sep 27, 2024

It could be a permissions issue with the es try making the deployment folder readable for everyone chmod 777 -R deployment and re start the container

@shaileshchaudhary11
Copy link
Author

shaileshchaudhary11 commented Sep 27, 2024

Thanks for reporting @shaileshchaudhary11 , would you mind providing the configuration profile you're using, so we can reproduce the issue on our side and investigate what may be happening? Thanks!

Profile : config.2406-gcs

I tried by giving permission: chmod 777 -R deployment
still getting same error

@shaileshchaudhary11
Copy link
Author

Hi @mbdebian, any updates on above issue, have you try to run it?

@remo87
Copy link
Contributor

remo87 commented Sep 30, 2024

You're right @shaileshchaudhary11 the permission change solves a different issue. I just tried cleaning my env and running the standalone with the same profile as you but it worked. I ran it in a Ubuntu 22.04.5 LTS which is the same version as yours. Could you please share console output to see if there's potential issues?

image

@shaileshchaudhary11
Copy link
Author

shaileshchaudhary11 commented Sep 30, 2024

Hii @remo87

Can you please tell me what commands did you run in sequence in the terminal?
I would like to try the exact same thing. Also, what did you mean by cleaning the env in the last response?

Also, Is there any other even a slight change in any file of the project that you made?

@remo87
Copy link
Contributor

remo87 commented Sep 30, 2024

sure thing!

 8281  2024-09-30 09:22:39 make platform_down
 8283  2024-09-30 09:23:00 sudo make clean
 8285  2024-09-30 09:23:52 make set_profile profile=2406-gcs
 8286  2024-09-30 09:24:09 make platform_up

I was testing with a different profile so I had different data and versions in my local environment. To test your configuration I ran make platform_down to destroy the containers and sudo make clean to delete the local files.

I did not changed the profile config.2406-gcs did a pull from master and my local repo has no changes.

@remo87
Copy link
Contributor

remo87 commented Oct 1, 2024

I forgot to mention that after the previous steps I also performed the chmod operation. after running the platform_up I end up with all healthy and running containers but opensearch (ot-es) doesn't work, then I apply the chmod and re start the container and then it works

@remo87
Copy link
Contributor

remo87 commented Oct 3, 2024

@shaileshchaudhary11 we've done a fix that might help you if you pull the latest version

@mbdebian
Copy link
Collaborator

mbdebian commented Oct 4, 2024

This has now been addressed, please, let us know in case you face any more issues.
Thanks!

@mbdebian mbdebian closed this as completed Oct 4, 2024
@shaileshchaudhary11
Copy link
Author

shaileshchaudhary11 commented Oct 12, 2024 via email

@mbdebian
Copy link
Collaborator

mbdebian commented Dec 2, 2024

@shaileshchaudhary11 , I noticed that you are using the configuration profile config.2406-gcs, this profile pulls the Elasticsearch and Clickhouse data volumes off of a Google Cloud Storage bucket where, although it's open to the public, it is set on requester pays, which may affect the capability of actually downloading the tarballs with the data for Elasticsearch and Clickhouse on your end.

This is also related to what's happening to @kenibrewer in #8 . And, actually, the file not found exception he mentions would make total sense.

Please, allow me to verify this hypothesis and I'll let you know soon.

Thanks!

@mbdebian mbdebian reopened this Dec 2, 2024
@mbdebian
Copy link
Collaborator

mbdebian commented Dec 2, 2024

I have reproduce this environment in Google Cloud, as we don't have yet an allocation in AWS, specs:

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy

Docker version 27.3.1, build ce12230
docker-compose version 1.29.2, build unknown

I've run this using the profile config.2406-gcs, which downloads the data volumes from our bucket in Google Cloud, and profile config.2406, which is the same, but the tarballs are downloaded from our EBI FTP location.

No JDK is needed, as everything is run via docker images.

When installing Docker, the regular user has been added to the docker group, so no root permission is needed for running the containers.

The changes can be found in branch local_issue_6_and_8.

@kenibrewer, would you mind trying this branch, with profile '2406' in your EC2 instance?

Thank you guys!

@kenibrewer
Copy link

Thanks @mbdebian . Your theory about the requester pays bucket seems solid to me. I'll test out the changes and let you know.

@kenibrewer
Copy link

Hi @mbdebian This fixed the problem for me, and I was able to get a working deployment. Thanks so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants