How to execute custodian policies using proxy ?

[cape-man]

Ask your question

I am trying to get custodian to execute an azure policy, but I cannot because the VM in which the policy is being executed doesn't have internet connectivity. But we do have a proxy to access the internet. Even after setting the proxy in VM , the azure policy is not getting executed. Is there a way to use proxy with custodian run command or the proxy parameters can be included in policy.yaml file?

Policy

---
policies:
- name: "busy-vms"
  resource: "azure.vm"
  filters:
  - type: "metric"
    metric: "Percentage CPU"
    op: "ge"
    aggregation: "average"
    threshold: 75
    timeframe: 24

Relevant log/traceback output

2021-12-13 10:14:51,709: custodian.commands:DEBUG Loaded file policy.yml. Contains 1 policies
2021-12-13 10:14:51,871: adal-python:ERROR 64839333-cbc0-4612-919a-c0dfc9f9a85c - OAuth2Client:Get Token request failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1808, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1540, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: SysCallError(104, 'ECONNRESET')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /85c997b9-f494-46b3-a11d-772983cf6f11/oauth2/token (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'ECONNRESET')",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/adal/oauth2_client.py", line 268, in get_token
    timeout=self._call_context.get('timeout', None))
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /85c997b9-f494-46b3-a11d-772983cf6f11/oauth2/token (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'ECONNRESET')",),))
2021-12-13 10:14:51,907: custodian.azure.session:ERROR Failed to authenticate.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1808, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1540, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: SysCallError(104, 'ECONNRESET')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /85c997b9-f494-46b3-a11d-772983cf6f11/oauth2/token (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'ECONNRESET')",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/capeuser/cloud-custodian/tools/c7n_azure/c7n_azure/session.py", line 145, in _initialize_session
    self._authenticate()
  File "/home/capeuser/cloud-custodian/tools/c7n_azure/c7n_azure/session.py", line 101, in _authenticate
    result = instance.authenticate()
  File "/home/capeuser/cloud-custodian/tools/c7n_azure/c7n_azure/session.py", line 413, in authenticate
    resource=self.resource_namespace)
  File "/usr/local/lib/python3.6/dist-packages/msrestazure/azure_active_directory.py", line 355, in __init__
    self.set_token()
  File "/usr/local/lib/python3.6/dist-packages/msrestazure/azure_active_directory.py", line 367, in set_token
    self.secret
  File "/usr/local/lib/python3.6/dist-packages/adal/authentication_context.py", line 179, in acquire_token_with_client_credentials
    return self._acquire_token(token_func)
  File "/usr/local/lib/python3.6/dist-packages/adal/authentication_context.py", line 128, in _acquire_token
    return token_func(self)
  File "/usr/local/lib/python3.6/dist-packages/adal/authentication_context.py", line 177, in token_func
    return token_request.get_token_with_client_credentials(client_secret)
  File "/usr/local/lib/python3.6/dist-packages/adal/token_request.py", line 310, in get_token_with_client_credentials
    token = self._oauth_get_token(oauth_parameters)
  File "/usr/local/lib/python3.6/dist-packages/adal/token_request.py", line 112, in _oauth_get_token
    return client.get_token(oauth_parameters)
  File "/usr/local/lib/python3.6/dist-packages/adal/oauth2_client.py", line 268, in get_token
    timeout=self._call_context.get('timeout', None))
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /85c997b9-f494-46b3-a11d-772983cf6f11/oauth2/token (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'ECONNRESET')",),))

[cape-man]

Please answer, as this issue has blocked the custodian from executing the policies.

[ajkerrigan]

Even after setting the proxy in VM , the azure policy is not getting executed.

By setting the proxy in the VM, do you mean tweaking the HTTP_PROXY / HTTPS_PROXY environment variables?

---

The best suggestion I have is to test/troubleshoot running the Azure CLI through your proxy first without custodian in the mix. Getting that working should sort custodian out also.

References:

• How to configure proxies for the Azure libraries
• Work behind a proxy

[cape-man]

yeah correct I was setting the environment variables,
export http_proxy="http://host:port"
export https_proxy="http://host:port"

[cape-man]

The issue is resolved by configuring the VPN(ZScaler) certificate into the server.