Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of Memory (Fatal) Error when Dropping Measurement #6796

Closed
goakley opened this issue Jun 7, 2016 · 2 comments
Closed

Out of Memory (Fatal) Error when Dropping Measurement #6796

goakley opened this issue Jun 7, 2016 · 2 comments
Assignees
Milestone

Comments

@goakley
Copy link

goakley commented Jun 7, 2016

Bug report

System info:
Debian GNU/Linux 8 (jessie)
3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) x86_64 GNU/Linux
InfluxDB 0.13.0

Steps to reproduce:

  1. Collect around 50,000 data points in a single measurement
  2. Attempt to drop the measurement with DROP MEASUREMENT [name]
  3. Wait a few seconds

Expected behavior:
After some time, InfluxDB should signify that it has successfully dropped the measurement, or that it has encountered some error and cannot finish dropping the measurement.

Actual behavior:
InfluxDB crashed and took a few minutes to restart from WAL segments. Running DROP MEASUREMENT [name] a second time succeeded.

Additional info: [Include gist of relevant config, logs, etc.]
Log contents (includes stack trace)

@jwilder jwilder added this to the 1.0.0 milestone Jun 8, 2016
@joelegasse
Copy link
Contributor

@goakley Can you see if @jwilder's change fixes your issue?

@jwilder
Copy link
Contributor

jwilder commented Jul 21, 2016

Going to close this as it should be fixed by #7015. The logs show hundreds of writes blocked on the tsdb.Store mutex. DropMeasurement incorrectly was holding the writing lock which locks up the whole tsdb.Store for writes and queries. The lock caused writes to pile up and ultimately a compaction ran and allocation triggered an OOM.

@jwilder jwilder closed this as completed Jul 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants