crash after restart with 0.5.0rc1

[schmurfy]

When restarting influxdb sometimes I get this:

2014-02-26T15:21:02Z.476760 [02/26/14 15:21:02] [INFO] Redirectoring logging to stdout
2014-02-26T15:21:02Z.476817 [02/26/14 15:21:02] [INFO] Starting Influx Server bound to 127.0.0.1 ...
2014-02-26T15:21:02Z.476850 [02/26/14 15:21:02] [INFO]
2014-02-26T15:21:02Z.476853 +---------------------------------------------+
2014-02-26T15:21:02Z.476854 |  _____        __ _            _____  ____   |
2014-02-26T15:21:02Z.476854 | |_   _|      / _| |          |  __ \|  _ \  |
2014-02-26T15:21:02Z.476855 |   | |  _ __ | |_| |_   ___  _| |  | | |_) | |
2014-02-26T15:21:02Z.476856 |   | | | '_ \|  _| | | | \ \/ / |  | |  _ <  |
2014-02-26T15:21:02Z.476856 |  _| |_| | | | | | | |_| |>  <| |__| | |_) | |
2014-02-26T15:21:02Z.476861 | |_____|_| |_|_| |_|\__,_/_/\_\_____/|____/  |
2014-02-26T15:21:02Z.476862 +---------------------------------------------+
2014-02-26T15:21:02Z.476863
2014-02-26T15:21:02Z.476863
2014-02-26T15:21:02Z.476875 [02/26/14 15:21:02] [INFO] Opening database at /apps/monitoring/data/metrics
2014-02-26T15:21:02Z.676486 panic: proto: required field "{Unknown}" not set
2014-02-26T15:21:02Z.676594
2014-02-26T15:21:02Z.676658 goroutine 1 [running]:
2014-02-26T15:21:02Z.676696 runtime.panic(0x851a20, 0xc21008fc10)
2014-02-26T15:21:02Z.676721     /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
2014-02-26T15:21:02Z.676747 main.main()
2014-02-26T15:21:02Z.676761     /tmp/influxdb-0.5.0-rc.1/src/daemon/influxd.go:104 +0x51b
2014-02-26T15:21:02Z.676793
2014-02-26T15:21:02Z.676811 goroutine 3 [chan receive]:
2014-02-26T15:21:02Z.676843 code.google.com/p/log4go.ConsoleLogWriter.run(0xc2100522c0, 0x7f39a11e30e8, 0xc210000008)
2014-02-26T15:21:02Z.676873     /tmp/influxdb-0.5.0-rc.1/src/code.google.com/p/log4go/termlog.go:27 +0x60
2014-02-26T15:21:02Z.676906 created by code.google.com/p/log4go.NewConsoleLogWriter
2014-02-26T15:21:02Z.676922     /tmp/influxdb-0.5.0-rc.1/src/code.google.com/p/log4go/termlog.go:19 +0x67
2014-02-26T15:21:02Z.676955
2014-02-26T15:21:02Z.676964 goroutine 4 [chan receive]:
2014-02-26T15:21:02Z.676986 code.google.com/p/log4go.ConsoleLogWriter.run(0xc210094160, 0x7f39a11e30e8, 0xc210000008)
2014-02-26T15:21:02Z.677022     /tmp/influxdb-0.5.0-rc.1/src/code.google.com/p/log4go/termlog.go:27 +0x60
2014-02-26T15:21:02Z.677048 created by code.google.com/p/log4go.NewConsoleLogWriter
2014-02-26T15:21:02Z.677062     /tmp/influxdb-0.5.0-rc.1/src/code.google.com/p/log4go/termlog.go:19 +0x67
2014-02-26T15:21:02Z.677095
2014-02-26T15:21:02Z.677104 goroutine 5 [syscall]:
2014-02-26T15:21:02Z.677133 runtime.goexit()
2014-02-26T15:21:02Z.677156     /usr/local/go/src/pkg/runtime/proc.c:1394
2014-02-26T15:21:02Z.677176
2014-02-26T15:21:02Z.677184 goroutine 6 [chan receive]:
2014-02-26T15:21:02Z.677214 datastore.(*LevelDbDatastore).periodicallyRotateRequestLog(0xc210077a00)
2014-02-26T15:21:02Z.677234     /tmp/influxdb-0.5.0-rc.1/src/datastore/leveldb_datastore.go:213 +0x41
2014-02-26T15:21:02Z.677260 created by datastore.NewLevelDbDatastore
2014-02-26T15:21:02Z.677275     /tmp/influxdb-0.5.0-rc.1/src/datastore/leveldb_datastore.go:205 +0x700

When this occurs I can restart as many times as I want it will stay this way until I remove the data folders and start from scratch, on this specific machine I get this error half the time when I restart, this makes it quite hard to keep any history xD

[pauldix]

On it

[elcct]

This happened to me as well. Deleting contents of /tmp related to InfluxDB helped.

[jvshahid]

@schmurfy is it possible to tar your data directory and send it over.

[pauldix]

@schmurfy actually, don't worry about it, I've managed to reproduce it locally. It has to do with the write buffer filling up and then attempting to replay from log. Should have a fix tomorrow. In the meantime you can set the write-buffer-size in storage to something high like 100k and it should be ok. Unless you're writing too much for LevelDB to keep up.

[schmurfy]

thanks for solving it so quickly :)

If I update my currently running influxdb with the fixed one, will the new one manage to "save" my database or will I need to delete it ?

[pauldix]

@schmurfy Because of the bugs in rc.1, you'll have to blow away your data directory and start fresh with rc.2