Fixing time limits aggregate queries and interval buckets

[jnutzmann]

History & Purpose

This pull request relates to two previous PR: #2016 and #2067.
#2016 was a PR I submitted to fix a bucket alignment issue. Unfortunately, I introduced a different bug that broke raw queries with a time range specified.
#2067 was created by @corylanou to fix my bug, but created another issue reported in #2120 and #2027.

In fixing this, I found another few issues (described below). IMO, they are less severe, so I will leave it up to the owners whether they should be fixed now with this PR or if new issues should be pulled and they should be fixed after.

Also, there is a definite need for more test cases around this.

Testing

There seems to be 3 types of queries that need to be tested:

1. Raw queries with a time range specified like: SELECT value FROM my_series WHERE time >= '2015-03-31T00:30:00Z' AND time < '2015-03-31T00:30:10Z' (with or without both time bounds)
2. Aggregate queries without a group by interval: SELECT count(value) FROM my_series WHERE time >= '2015-03-31T00:30:00Z' AND time < '2015-03-31T00:30:10Z'
3. Aggregate queries with a group by interval: SELECT count(value) FROM my_series WHERE time >= '2015-03-31T00:30:00Z' GROUP BY time(1h)

To verify these are now correct, I created a database with data points spaced every 2 seconds for a few days.

Here are my findings case-by-case:

Case 1

Query: SELECT value FROM my_series WHERE time >= '2015-03-31T00:30:00Z' AND time <= '2015-03-31T00:30:10Z'

Results (same on master vs this PR):

{
    "results": [
        {
            "series": [
                {
                    "name": "my_series",
                    "columns": [
                        "time",
                        "value"
                    ],
                    "values": [
                        [
                            "2015-03-31T00:30:00Z",
                            79154
                        ],
                        [
                            "2015-03-31T00:30:02Z",
                            79156
                        ],
                        [
                            "2015-03-31T00:30:04Z",
                            79158
                        ],
                        [
                            "2015-03-31T00:30:06Z",
                            79160
                        ],
                        [
                            "2015-03-31T00:30:08Z",
                            79162
                        ]
                    ]
                }
            ]
        }
    ]
}

New Issue 1: The results are missing the last point at 2015-03-31T00:30:10Z. If we run the query with time < '2015-03-31T00:30:10Z' (getting rid of equal), we get the same result. If we add a second (time < '2015-03-31T00:30:11Z'), then we get the point back. The top bound is probably off by a nanosecond.

Case 2

Query1: SELECT count(value) FROM my_series WHERE time >= '2015-03-31T00:30:00Z' AND time < '2015-03-31T00:30:10Z'

Results (same for master and PR):

{
    "results": [
        {
            "series": [
                {
                    "name": "my_series",
                    "columns": [
                        "time",
                        "count"
                    ],
                    "values": [
                        [
                            "2015-03-31T00:30:00Z",
                            5
                        ]
                    ]
                }
            ]
        }
    ]
}

Query 2: SELECT count(value) FROM my_series WHERE time >= '2015-03-31T00:30:00Z'

This query is where the bug fix comes in. In master we get inconsistent values (0, 2760, 17392, 37953, etc). Timestamps varied as well. With this PR, we get:

{
    "results": [
        {
            "series": [
                {
                    "name": "my_series",
                    "columns": [
                        "time",
                        "count"
                    ],
                    "values": [
                        [
                            "2015-03-31T00:30:00Z",
                            37953
                        ]
                    ]
                }
            ]
        }
    ]
}

As expected, the PR fixes this issue.

Case 3

Query: SELECT count(value) FROM my_series WHERE time >= '2015-03-31T00:30:00Z' AND time < '2015-03-31T4:30:00Z' GROUP BY time(1h)

Results (same in master and PR):

{
    "results": [
        {
            "series": [
                {
                    "name": "my_series",
                    "columns": [
                        "time",
                        "count"
                    ],
                    "values": [
                        [
                            "2015-03-31T00:00:00Z",
                            900
                        ],
                        [
                            "2015-03-31T01:00:00Z",
                            1800
                        ],
                        [
                            "2015-03-31T02:00:00Z",
                            1800
                        ],
                        [
                            "2015-03-31T03:00:00Z",
                            1800
                        ],
                        [
                            "2015-03-31T04:00:00Z",
                            1800
                        ]
                    ]
                }
            ]
        }
    ]
}

New Issue 2: In the last time interval we get all the results back, when we should only be getting half, as we specified 2015-03-31T4:30:00Z as the end time. We can vary this from 2015-03-31T4:01:00Z to 2015-03-31T5:00:00Z and get the same results.

Please let me know your thoughts on this (and review my changes carefully :) )

[jnutzmann]

My tests failed the race step. I don't think I changed anything that would affect this? Perhaps someone has seen this before and could help?

[raft] log2 2015/04/02 04:34:52 log state change: follower => stopped (term=1)
[raft] log2 2015/04/02 04:34:52 closed
--- FAIL: TestCluster_Apply (0.07s)
    log_test.go:310: unexpected command count(2): 0
=== RUN TestLog_Elect
Log //log0: 0xc208036680
Log //log1: 0xc208036b60
Log //log2: 0xc208036d00

[raft] log0 2015/04/02 04:34:52 Open: fsm: index=0
[raft] log0 2015/04/02 04:34:52 log pending: waiting for initialization or join
[raft] log0 2015/04/02 04:34:52 changing term: 0 => 1

[otoolep]

@jnutzmann -- this is a known race in our code, which we're working on. Let me kick your build.

[toddboom]

@pauldix review, por favor?

[otoolep]

Needs a rebase too, let me know if you need assistance @jnutzmann

[jnutzmann]

Rebase is done. Looks like it failed in the race again, might need a build kick. I don't think I can do that?

[toddboom]

@jnutzmann kicking it now

[toddboom]

@jnutzmann thanks for the rebase and continued effort. we've got one more branch we need to get merged in before this gets accepted - i'll ping you once that's in.

[jnutzmann]

I'm closing this, as it has been obsoleted by other, larger changes.

An update on the issues in it:

New issue 1 - Fixed.
New issue 2 - Issue opened for it #2347

#2120 / #2027 - The value portion of these are fixed, but the timestamp still varies with system time.