cosbench fails with Unable to execute HTTP request: The target server failed to respond

[davidbw246]

Environment info

• NooBaa Version: noobaa-core:master-20210510
• Platform: ocp 4.7.4

Actual behavior

1. All cosbench ops to my ocp+ cluster fail.

Expected behavior

1. cosbench ops should succeed.

Steps to reproduce

1. Setup cosbench with creds/enpoint for ocp+ cluster with noobaa-core:5.8.0-20210418 or 20210510 and attempt any ops.

More information

oc logs noobaa-endpoint does not log any records for the failures.

cosbench logs show: 

• 10.17.83.245:30855 endpoint
  - Unable to execute HTTP request: The target server failed to respond
•  s3-openshift-storage.apps.cosbenchplus.cp.fyre.ibm.com endpoint 
  - Status Code: 503 AWS Error Code: Service unavailable
• endpoint=http://10.17.83.245 endpoint
  - Status Code: 503 AWS Error Code: Service unavailable

[guymguym]

@davidbw246 Hi. Does any other S3 access succeed from the same host and creds that you're running cosbench? Is this a network issue perhaps? Not enough info to know, but I would start by checking that aws cli is able to connect:

AWS_ACCESS_KEY_ID=??? AWS_SECRET_ACCESS_KEY=??? aws s3 --endpoint ??? ls

[davidbw246]

Hi @guymguym . Yes, s3 commands(mb, cp, etc) succeed from the same host and creds that I am running cosbench on.

[davidbw246]

cosbench is working now on my ocp+ cluster with http.
I had been using the port that "noobaa status" reported under NodePorts, which looks like is for port 443 which is for https.
Greg Kishi pointed out should use the first port listed from "oc get service s3", which is for port 80 and is for http.

cosbench with https with port 443 fails with "Unable to execute HTTP request: peer not authenticated", possibly due to a known issue with cosbench not working with self signed certificates(intel-cloud/cosbench#277).

[nimrod-becker]

@davidbw246 does this mean this issue is closed ?

[davidbw246]

Yes, issue is closed.