Postgres DB connection limit exceeded

[chouseknecht]

Seeing the following traceback in our logs:

2018-07-27 11:04:57,741 ERROR exception: Internal Server Error: /api/v1/search/cloud_platforms/
--
  | Traceback (most recent call last):
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 131, in get_response
  | response = middleware_method(request, response)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/contrib/sessions/middleware.py", line 47, in process_response
  | if request.session.get_expire_at_browser_close():
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 292, in get_expire_at_browser_close
  | if self.get('_session_expiry') is None:
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 68, in get
  | return self._session.get(key, default)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 207, in _get_session
  | self._session_cache = self.load()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py", line 35, in load
  | expire_date__gt=timezone.now()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
  | return getattr(self.get_queryset(), name)(*args, **kwargs)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
  | num = len(clone)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
  | self._fetch_all()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
  | self._result_cache = list(self._iterable_class(self))
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
  | results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
  | cursor = self.connection.cursor()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
  | return self._cursor()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
  | self.ensure_connection()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
  | self.connect()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
  | six.reraise(dj_exc_type, dj_exc_value, traceback)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
  | self.connect()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
  | self.connection = self.get_new_connection(conn_params)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
  | connection = Database.connect(**conn_params)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
  | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
  | OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser and rds_superuser connections

[chouseknecht]

The problem seems to have gone away. The question is, how do we avoid this in the future?

[juliedavila]

@chouseknecht is it on RDS? If so, the it's an easy to tweak via RDS Parameter Groups. https://stackoverflow.com/questions/38555679/aws-rds-postgresql-error-remaining-connection-slots-are-reserved-for-non-replic

[chouseknecht]

@defionscode

Thanks for the suggestion. I think we have this fixed now in Django settings.