+# 
+
[](https://travis-ci.org/Comcast/trickster/)
[](https://goreportcard.com/report/github.com/Comcast/trickster)
-[](https://coveralls.io/github/Comcast/trickster?branch=master)
-[](https://bestpractices.coreinfrastructure.org/en/projects/2518)
+[](https://coveralls.io/github/Comcast/trickster?branch=next)
+[](https://bestpractices.coreinfrastructure.org/en/projects/2518)
+[](https://hub.docker.com/r/tricksterio/trickster)
+[](https://twitter.com/tricksterio)
+
+Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases.
+
+
+
+## HTTP Reverse Proxy Cache
+
+Trickster is a fully-featured HTTP Reverse Proxy Cache for HTTP applications like static file servers and web API's.
+
+### Trickster Feature Highlights
+
+* [Supports TLS](./docs/tls.md) frontend termination and backend origination
+* Offers several options for a [caching layer](./docs/caches.md), including in-memory, filesystem, Redis and bbolt
+* [Highly customizable](./docs/configuring.md), using simple configuration settings, [down to the HTTP Path](./docs/paths.md)
+* Built-in Prometheus [metrics](./docs/metrics.md) and customizable [Health Check](./docs/health.md) Endpoints for end-to-end monitoring
+* [Negative Caching](./docs/negative-caching.md) to prevent domino effect outages
+* High-performance [Collapsed Forwarding](./docs/collapsed-forwarding.md)
+* Best-in-class [Range Request caching and acceleration](./docs/range_request.md).
+
+## Dashboard Acceleration
+
+Trickster dramatically improves dashboard chart rendering times for end users, while eliminating redundant computations on the TSDBs it fronts. In short, Trickster makes read-heavy Dashboard/TSDB environments, as well as those with highly-cardinalized datasets, significantly more performant and scalable.
-Trickster is a reverse proxy cache for the [Prometheus](https://github.com/prometheus/prometheus) [HTTP APIv1](https://prometheus.io/docs/prometheus/latest/querying/api/) that dramatically accelerates dashboard rendering times for any series queried from Prometheus.
+## Compatibility
-#### NOTE: Trickster is currently actively developed under the [next](https://github.com/Comcast/trickster/tree/next) Branch for our milestone 1.0 Release.
+Trickster works with virtually any Dashboard application that makes queries to any of these TSDB's:
-
+
Prometheus
-## How it works
+
ClickHouse
+
+
InfluxDB
+
+
Circonus IRONdb
+
+See the [Supported Origin Types](./docs/supported-origin-types.md) document for full details
+
+## How Trickster Accelerates Time Series
### 1. Delta Proxy
-Most dashboards request the entire time range of data from the time series database, every time a dashboard loads or reloads. Trickster's Delta Proxy inspects the time range of a client query to determine what data points are already cached, and requests from Prometheus only the data points still needed to service the client request. This results in dramatically faster chart load times for everyone, since Prometheus is queried only for tiny incremental changes on each dashboard load, rather than several hundred data points of duplicative data.
+
+Most dashboards request from a time series database the entire time range of data they wish to present, every time a user's dashboard loads, as well as on every auto-refresh. Trickster's Delta Proxy inspects the time range of a client query to determine what data points are already cached, and requests from the tsdb only the data points still needed to service the client request. This results in dramatically faster chart load times for everyone, since the tsdb is queried only for tiny incremental changes on each dashboard load, rather than several hundred data points of duplicative data.
### 2. Step Boundary Normalization
-When Trickster requests data from Prometheus, it adjusts the clients's requested time range slightly to ensure that all data points returned by Prometheus are aligned to normalized step boundaries. For example, if the step is 300s, all data points will fall on the clock 0's and 5's. This ensures that the data is highly cacheable, is conveyed visually to users in a more familiar way, and that all dashboard users see identical data on their screens.
+
+When Trickster requests data from a tsdb, it adjusts the clients's requested time range slightly to ensure that all data points returned are aligned to normalized step boundaries. For example, if the step is 300s, all data points will fall on the clock 0's and 5's. This ensures that the data is highly cacheable, is conveyed visually to users in a more familiar way, and that all dashboard users see identical data on their screens.
### 3. Fast Forward
+
Trickster's Fast Forward feature ensures that even with step boundary normalization, real-time graphs still always show the most recent data, regardless of how far away the next step boundary is. For example, if your chart step is 300s, and the time is currently 1:21p, you would normally be waiting another four minutes for a new data point at 1:25p. Trickster will break the step interval for the most recent data point and always include it in the response to clients requesting real-time data.
@@ -52,7 +88,6 @@ binary into your `GOPATH`:
$ go get github.com/Comcast/trickster
$ trickster -origin http://prometheus.example.com:9090
-
You can also clone the repository yourself and build using `make`:
$ mkdir -p $GOPATH/src/github.com/Comcast
@@ -62,25 +97,26 @@ You can also clone the repository yourself and build using `make`:
$ make build
$ ./trickster -origin http://prometheus.example.com:9090
-The Makefile provides several targets:
+The Makefile provides several targets, including:
- * *build*: build the `trickster` binary
- * *docker*: build a docker container for the current `HEAD`
- * *clean*: delete previously-built binaries and object files
+* *build*: build the `trickster` binary
+* *docker*: build a docker container for the current `HEAD`
+* *clean*: delete previously-built binaries and object files
+* *test*: runs unit tests
## More information
- * Refer to the docs directory for additional info.
+* Refer to the [docs](./docs/) directory for additional info.
## Contributing
Refer to [CONTRIBUTING.md](CONTRIBUTING.md)
-## Who uses Trickster?
+## Who Is Using Trickster
As the Trickster community grows, we'd like to keep track of who is using it in their stack. We invite you to submit a PR with your company name and @githubhandle to be included on the list.
1. [Comcast](https://comcast.github.io) [[@jranson](https://github.com/jranson)]
2. [Selfnet e.V.](https://www.selfnet.de/) [[@ThoreKr](https://github.com/ThoreKr)]
3. [swarmstack](https://github.com/swarmstack) [[@mh720](https://github.com/mh720)]
-4. [Hostinger](https://www.hostinger.com/) [[@ton31337](https://github.com/ton31337)]
\ No newline at end of file
+4. [Hostinger](https://www.hostinger.com/) [[@ton31337](https://github.com/ton31337)]
diff --git a/boltdb.go b/boltdb.go
deleted file mode 100644
index 2c2fe7488..000000000
--- a/boltdb.go
+++ /dev/null
@@ -1,232 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "fmt"
- "strconv"
- "strings"
- "time"
-
- bolt "github.com/coreos/bbolt"
- "github.com/go-kit/kit/log/level"
-)
-
-// BoltDBCache describes a BoltDB Cache
-type BoltDBCache struct {
- T *TricksterHandler
- Config BoltDBCacheConfig
- dbh *bolt.DB
-}
-
-// Connect instantiates the BoltDBCache mutex map and starts the Expired Entry Reaper goroutine
-func (c *BoltDBCache) Connect() error {
- level.Info(c.T.Logger).Log("event", "boltdb cache setup", "cacheFile", c.Config.Filename)
-
- var err error
- c.dbh, err = bolt.Open(c.Config.Filename, 0644, &bolt.Options{Timeout: 1 * time.Second})
- if err != nil {
- return err
- }
-
- err = c.dbh.Update(func(tx *bolt.Tx) error {
- tx.CreateBucketIfNotExists([]byte(c.Config.Bucket))
- if err != nil {
- return fmt.Errorf("create bucket: %s", err)
- }
- return nil
- })
- if err != nil {
- return err
- }
-
- go c.Reap()
- return nil
-}
-
-// Store places an object in the cache using the specified key and ttl
-func (c *BoltDBCache) Store(cacheKey string, data string, ttl int64) error {
-
- expKey, dataKey := c.getKeyNames(cacheKey)
- expiration := []byte(strconv.FormatInt(time.Now().Unix()+ttl, 10))
-
- err := c.dbh.Update(func(tx *bolt.Tx) error {
-
- b := tx.Bucket([]byte(c.Config.Bucket))
-
- err := b.Put([]byte(dataKey), []byte(data))
- if err != nil {
- return err
- }
-
- return b.Put([]byte(expKey), expiration)
- })
- if err != nil {
- return err
- }
-
- level.Debug(c.T.Logger).Log("event", "boltdb cache store", "key", dataKey, "expKey", expKey)
-
- return nil
-}
-
-// Retrieve looks for an object in cache and returns it (or an error if not found)
-func (c *BoltDBCache) Retrieve(cacheKey string) (string, error) {
-
- level.Debug(c.T.Logger).Log("event", "boltdb cache retrieve", "key", cacheKey)
-
- _, dataKey := c.getKeyNames(cacheKey)
-
- c.checkExpiration(cacheKey)
-
- return c.retrieve(dataKey)
-}
-
-// retrieve looks for an object in cache and returns it (or an error if not found)
-func (c *BoltDBCache) retrieve(cacheKey string) (string, error) {
-
- content := ""
-
- err := c.dbh.View(func(tx *bolt.Tx) error {
- b := tx.Bucket([]byte(c.Config.Bucket))
- v := b.Get([]byte(cacheKey))
- if v == nil {
- level.Debug(c.T.Logger).Log("event", "boltdb cache miss", "key", cacheKey)
- return fmt.Errorf("Value for key [%s] not in cache", cacheKey)
- }
- content = string(v)
- return nil
- })
- if err != nil {
- return "", err
- }
-
- return content, nil
-}
-
-// checkExpiration verifies that a cacheKey is not expired
-func (c *BoltDBCache) checkExpiration(cacheKey string) {
-
- expKey, _ := c.getKeyNames(cacheKey)
-
- content, err := c.retrieve(expKey)
- if err == nil {
- // We found this key, let's see if it's expired
- expiration, err := strconv.ParseInt(string(content), 10, 64)
- if err != nil || expiration < time.Now().Unix() {
- c.Delete(cacheKey)
- }
- }
-}
-
-// Delete removes an object in cache, if present
-func (c *BoltDBCache) Delete(cacheKey string) error {
-
- level.Debug(c.T.Logger).Log("event", "boltdb cache delete", "key", cacheKey)
-
- expKey, dataKey := c.getKeyNames(cacheKey)
-
- return c.dbh.Update(func(tx *bolt.Tx) error {
-
- b := tx.Bucket([]byte(c.Config.Bucket))
-
- err1 := b.Delete([]byte(expKey))
- if err1 != nil {
- level.Error(c.T.Logger).Log("event", "boltdb cache key delete failure", "key", expKey, "reason", err1.Error())
- }
-
- err2 := b.Delete([]byte(dataKey))
- if err2 != nil {
- level.Error(c.T.Logger).Log("event", "boltdb cache key delete failure", "key", dataKey, "reason", err2.Error())
- }
-
- c.T.ChannelCreateMtx.Lock()
-
- // Close out the channel if it exists
- if _, ok := c.T.ResponseChannels[cacheKey]; ok {
- close(c.T.ResponseChannels[cacheKey])
- delete(c.T.ResponseChannels, cacheKey)
- }
-
- // Unlock
- c.T.ChannelCreateMtx.Unlock()
-
- if err1 != nil {
- return err1
- }
- if err2 != nil {
- return err2
- }
-
- return nil
-
- })
-
-}
-
-// Reap continually iterates through the cache to find expired elements and removes them
-func (c *BoltDBCache) Reap() {
-
- for {
- c.ReapOnce()
- time.Sleep(time.Duration(c.T.Config.Caching.ReapSleepMS) * time.Millisecond)
- }
-
-}
-
-// ReapOnce makes a single iteration through the cache to to find and remove expired elements
-func (c *BoltDBCache) ReapOnce() {
-
- now := time.Now().Unix()
- expiredKeys := make([]string, 0)
-
- // Iterate through the cache to find any expiration keys and check their value
- c.dbh.View(func(tx *bolt.Tx) error {
- // Assume bucket exists and has keys
- b := tx.Bucket([]byte(c.Config.Bucket))
- cursor := b.Cursor()
-
- for k, v := cursor.First(); k != nil; k, v = cursor.Next() {
-
- expKey := string(k)
-
- if strings.HasSuffix(expKey, ".expiration") {
-
- expiration, err := strconv.ParseInt(string(v), 10, 64)
- if err != nil || expiration < now {
-
- expiredKeys = append(expiredKeys, strings.Replace(expKey, ".expiration", "", -1))
-
- }
- }
- }
-
- return nil
- })
-
- // Iterate through the expired keys so we can delete them
- for _, cacheKey := range expiredKeys {
- c.Delete(cacheKey)
- }
-
-}
-
-// Close closes the BoltDBCache
-func (c *BoltDBCache) Close() error {
- return c.dbh.Close()
-}
-
-func (c *BoltDBCache) getKeyNames(cacheKey string) (string, string) {
- return cacheKey + ".expiration", cacheKey + ".data"
-}
diff --git a/boltdb_test.go b/boltdb_test.go
deleted file mode 100644
index ff591e81a..000000000
--- a/boltdb_test.go
+++ /dev/null
@@ -1,104 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "testing"
-
- "github.com/go-kit/kit/log"
-)
-
-func TestBoltDBCache_Connect(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- bc := BoltDBCache{T: &tr, Config: BoltDBCacheConfig{Filename: "/tmp/test.db", Bucket: "trickster_test"}}
-
- // it should connect
- err := bc.Connect()
- if err != nil {
- t.Error(err)
- }
-
- bc.Close()
-
-}
-
-func TestBoltDBCache_Store(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- bc := BoltDBCache{T: &tr, Config: BoltDBCacheConfig{Filename: "/tmp/test.db", Bucket: "trickster_test"}}
-
- err := bc.Connect()
- if err != nil {
- t.Error(err)
- }
- defer bc.Close()
-
- // it should store a value
- err = bc.Store("cacheKey", "data", 60000)
- if err != nil {
- t.Error(err)
- }
-}
-
-func TestBoltDBCache_Delete(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- bc := BoltDBCache{T: &tr, Config: BoltDBCacheConfig{Filename: "/tmp/test.db", Bucket: "trickster_test"}}
-
- err := bc.Connect()
- if err != nil {
- t.Error(err)
- }
- defer bc.Close()
-
- // it should store a value
- err = bc.Store("cacheKey", "data", 60000)
- if err != nil {
- t.Error(err)
- }
-
- // it should store a value
- err = bc.Delete("cacheKey")
- if err != nil {
- t.Error(err)
- }
-
-}
-
-func TestBoltDBCache_Retrieve(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- bc := BoltDBCache{T: &tr, Config: BoltDBCacheConfig{Filename: "/tmp/test.db", Bucket: "trickster_test"}}
-
- err := bc.Connect()
- if err != nil {
- t.Error(err)
- }
- defer bc.Close()
-
- err = bc.Store("cacheKey", "data", 60000)
- if err != nil {
- t.Error(err)
- }
-
- // it should retrieve a value
- data, err := bc.Retrieve("cacheKey")
- if err != nil {
- t.Error(err)
- }
- if data != "data" {
- t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
- }
-}
diff --git a/cache.go b/cache.go
deleted file mode 100644
index 8fe6b6e00..000000000
--- a/cache.go
+++ /dev/null
@@ -1,51 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "fmt"
-)
-
-const (
- // Cache interface types
- ctMemory = "memory"
- ctFilesystem = "filesystem"
- ctRedis = "redis"
- ctBoltDB = "boltdb"
-)
-
-// Cache is the interface for the supported caching fabrics
-// When making new cache types, Retrieve() must return an error on cache miss
-type Cache interface {
- Connect() error
- Store(cacheKey string, data string, ttl int64) error
- Retrieve(cacheKey string) (string, error)
- Reap()
- Close() error
-}
-
-func getCache(t *TricksterHandler) Cache {
- switch t.Config.Caching.CacheType {
- case ctFilesystem:
- return &FilesystemCache{Config: t.Config.Caching.Filesystem, T: t}
- case ctBoltDB:
- return &BoltDBCache{Config: t.Config.Caching.BoltDB, T: t}
- case ctRedis:
- return &RedisCache{Config: t.Config.Caching.Redis, T: t}
- case ctMemory:
- return &MemoryCache{T: t}
- default:
- panic(fmt.Errorf("Invalid cache type: %q", t.Config.Caching.CacheType))
- }
-}
diff --git a/logging_test.go b/cmd/promsim/main.go
similarity index 58%
rename from logging_test.go
rename to cmd/promsim/main.go
index 59efa6016..bd7e4a95e 100644
--- a/logging_test.go
+++ b/cmd/promsim/main.go
@@ -14,26 +14,24 @@
package main
import (
- "testing"
+ "fmt"
+ "net/http"
+ "os"
+
+ "github.com/Comcast/trickster/pkg/promsim"
)
-func TestNewLogger(t *testing.T) {
- testCases := []string{
- "debug",
- "info",
- "warn",
- "error",
- "none",
- }
- // it should create a logger for each level
- for _, tc := range testCases {
- t.Run(tc, func(t *testing.T) {
- newLogger(LoggingConfig{LogLevel: tc}, tc)
- })
+func main() {
+
+ port := "9090"
+ if len(os.Args) > 1 && os.Args[1] != "" {
+ port = os.Args[1]
}
-}
-func TestNewLogger_LogFile(t *testing.T) {
- // it should create a logger that outputs to a log file ("out.test.log")
- newLogger(LoggingConfig{LogFile: "out.log"}, "test")
+ fmt.Println("Starting up PromSim on port", port)
+ err := http.ListenAndServe(fmt.Sprintf(":%s", port), promsim.MuxWithRoutes())
+ if err != nil {
+ fmt.Println(err.Error())
+ os.Exit(0)
+ }
}
diff --git a/cmd/rangesim/main.go b/cmd/rangesim/main.go
new file mode 100644
index 000000000..199d9d6a5
--- /dev/null
+++ b/cmd/rangesim/main.go
@@ -0,0 +1,38 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package main
+
+import (
+ "fmt"
+ "net/http"
+ "os"
+
+ "github.com/Comcast/trickster/pkg/rangesim"
+)
+
+func main() {
+
+ port := "8090"
+ if len(os.Args) > 1 && os.Args[1] != "" {
+ port = os.Args[1]
+ }
+
+ fmt.Println("Starting up RangeSim on port", port)
+
+ err := http.ListenAndServe(fmt.Sprintf(":%s", port), rangesim.MuxWithRoutes())
+ if err != nil {
+ fmt.Println(err.Error())
+ os.Exit(0)
+ }
+}
diff --git a/cmd/trickster/conf/example.conf b/cmd/trickster/conf/example.conf
new file mode 100644
index 000000000..7458e7bec
--- /dev/null
+++ b/cmd/trickster/conf/example.conf
@@ -0,0 +1,435 @@
+#
+# Trickster 1.0 Example Configuration File - Exhaustive
+#
+# To use this, run: trickster -config /path/to/example.conf
+#
+# This file contains descriptions and examples for all
+# Trickster configuration options. More documentation is
+# available at https://github.com/Comcast/trickster/docs/
+#
+# Optional configs are commented out, required configs are uncommented
+# and set to common values that let you try it out with Prometheus
+#
+# Copyright 2018 Comcast Cable Communications Management, LLC
+#
+
+# [main]
+
+## instance_id allows you to run multiple Trickster processes on the same host and log to separate files
+## Useful for baremetal, not so much for elastic deployments, so only uncomment if you really need it
+## default is 0, which means ignored
+#instance_id = 0
+
+## config_handler_path provides the HTTP path to view a read-only printout of the running configuration
+## which can be reached at http://your-trickster-endpoint:port/$config_handler_path
+## default is '/trickster/config'
+# config_handler_path = '/trickster/config'
+
+## ping_handler_path provides the HTTP path you will use to perform an uptime health check against Trickster
+## which can be reached at http://your-trickster-endpoint:port/$ping_handler_path
+## default is '/trickster/ping'
+# ping_handler_path = '/trickster/ping'
+
+
+# Configuration options for the Trickster Frontend
+[frontend]
+
+# listen_port defines the port on which Trickster's Front-end HTTP Proxy server listens.
+listen_port = 9090
+
+## listen_address defines the ip on which Trickster's Front-end HTTP Proxy server listens.
+## empty by default, listening on all interfaces
+# listen_address = ''
+
+## tls_listen_address defines the ip on which Trickster's Front-end TLS Proxy server listens.
+## empty by default, listening on all interfaces
+# tls_listen_address = ''
+
+## tls_listen_port defines the port on which Trickster's Front-end TLS Proxy server listens.
+## The default is 0, which means TLS is not used, even if certificates are configured below.
+# tls_listen_port = 0
+
+## connections_limit defines the maximum number of concurrent connections
+## Trickster's Proxy server may handle at any time.
+## 0 by default, unlimited.
+# connections_limit = 0
+
+# [caches]
+
+ # [caches.default]
+ ## cache_type defines what kind of cache Trickster uses
+ ## options are 'bbolt', 'badger', 'filesystem', 'memory', and 'redis'
+ ## The default is 'memory'.
+ # cache_type = 'memory'
+
+ ### Configuration options for the Cache Index
+ ## The Cache Index handles key management and retention for bbolt, filesystem and memory
+ ## Redis and BadgerDB handle those functions natively and does not use the Trickster's Cache Index
+ # [caches.default.index]
+
+ ## reap_interval_secs defines how long the Cache Index reaper sleeps between reap cycles. Default is 3 (3s)
+ # reap_interval_secs = 3
+
+ ## flush_interval_secs sets how often the Cache Index saves its metadata to the cache from application memory. Default is 5 (5s)
+ # flush_interval_secs = 5
+
+ ## max_size_bytes indicates how large the cache can grow in bytes before the Index evicts least-recently-accessed items. default is 512MB
+ # max_size_bytes = 536870912
+
+ ## max_size_backoff_bytes indicates how far below max_size_bytes the cache size must be to complete a byte-size-based eviction exercise. default is 16MB
+ # max_size_backoff_bytes = 16777216
+
+ ## max_size_objects indicates how large the cache can grow in objects before the Index evicts least-recently-accessed items. default is 0 (infinite)
+ # max_size_objects = 0
+
+ ## max_size_backoff_objects indicates how far under max_size_objects the cache size must be to complete object-size-based eviction exercise. default is 100
+ # max_size_backoff_objects = 100
+
+ ### Configuration options when using a Redis Cache
+ # [caches.default.redis]
+
+ ## client_type indicates which kind of Redis client to use. Options are: 'standard', 'cluster' and 'sentinel'
+ ## default is 'standard'
+ # client_type = 'standard'
+
+ ### Supported by Redis (standard) #####################################
+ ### These configurations are ignored by Redis Sentinel and Redis Cluster
+ ###
+
+ ## endpoint defines the fqdn+port or path to a unix socket file for connecting to redis
+ ## default is 'redis:6379'
+ # endpoint = 'redis:6379'
+ #
+
+ ### Supported by Redis Cluster and Redis Sentinel #####################
+ ### These conigurations are ignored by Redis (standard)
+ ###
+
+ ## endpoints is used for Redis Cluster and Redis Sentinel to define a list of endpoints
+ ## default is ['redis:6379']
+ # endpoints = ['redis:6379']
+ #
+
+ ### Supported by Redis Sentinel #######################################
+ ### These conigurations are ignored by Redis (standard) and Redis Cluster
+ ###
+ ## sentinel_master should be set when using Redis Sentinel to indicate the Master Node
+ ## sentinel_master = ''
+ #
+
+ ### Supported by all Redis Client Types ###############################
+ ### See the go-redis documentation at https://github.com/go-redis/redis/blob/master/options.go
+ ### for more information on tuning these settings
+
+ ## protocol defines the protocol for connecting to redis ('unix' or 'tcp'). 'tcp' is default
+ # protocol = 'tcp'
+
+ ## password provides the redis password. default is empty string ''
+ # password = ''
+
+ ## db is the Database to be selected after connecting to the server. default is 0
+ # db = 0
+
+ ## max_retries is the maximum number of retries before giving up on the command
+ # max_retries = 0
+
+ ## min_retry_backoff_ms is the minimum backoff time between each retry
+ # min_retry_backoff_ms = 8
+
+ ## max_retry_backoff_ms is the maximum backoff time between each retry
+ # max_retry_backoff_ms = 512
+
+ ## dial_timeout_ms is the timeout for establishing new connections
+ # dial_timeout_ms = 5000
+
+ ## read_timeout_ms is the timeout for socket reads. If reached, commands will fail with a timeout instead of blocking.
+ # read_timeout_ms = 3000
+
+ ## write_timeout_ms is the timeout for socket writes. If reached, commands will fail with a timeout instead of blocking.
+ # write_timeout_ms = 3000
+
+ ## pool_size is the maximum number of socket connections.
+ # pool_size = 20
+
+ ## min_idle_conns is the minimum number of idle connections which is useful when establishing new connection is slow.
+ # min_idle_conns = 0
+
+ ## max_conn_age_ms is the connection age at which client retires (closes) the connection.
+ # max_conn_age_ms = 0
+
+ ## pool_timeout_ms is the amount of time client waits for connection if all connections are busy before returning an error.
+ # pool_timeout_ms = 4000
+
+ ## idle_timeout_ms is the amount of time after which client closes idle connections.
+ # idle_timeout_ms = 300000
+
+ ## idle_check_frequency_ms is the frequency of idle checks made by idle connections reaper.
+ # idle_check_frequency_ms = 60000
+
+
+ ### Configuration options when using a Filesystem Cache ###############
+ # [caches.default.filesystem]
+ ## cache_path defines the directory location under which the Trickster cache will be maintained
+ ## default is '/tmp/trickster'
+ # cache_path = '/tmp/trickster'
+
+ ### Configuration options when using a bbolt Cache ####################
+ # [caches.default.bbolt]
+
+ ## filename defines the file where the Trickster cache will be maintained
+ ## default is 'trickster.db'
+ # filename = 'trickster.db'
+
+ ## bucket defines the name of the bbolt bucket (similar to a namespace) under which our key value store lives
+ ## default is 'trickster'
+ # bucket = 'trickster'
+
+ ### Configuration options when using a Badger cache ###################
+ # [caches.default.badger]
+ ## directory defines the directory location under which the Badger data will be maintained
+ ## default is '/tmp/trickster'
+ # directory = '/tmp/trickster'
+ ## value_directory defines the directory location under which the Badger value log will be maintained
+ ## default is '/tmp/trickster'
+ # value_directory = '/tmp/trickster'
+
+ ## Example of a second cache, sans comments, that origin configs below could use with: cache_name = 'bbolt_example'
+ #
+ # [caches.bbolt_example]
+ # cache_type = 'bbolt'
+
+ # [caches.bbolt_example.bbolt]
+ # filename = 'trickster.db'
+ # bucket = 'trickster'
+
+ # [caches.bbolt_example.index]
+ # reap_interval_secs = 3
+ # flush_interval_secs = 5
+ # max_size_bytes = 536870912
+ # size_backoff_bytes = 16777216
+
+## Negative Caching Configurations
+## A Negative Cache is a map of HTTP Status Codes that are cached for the specified duration,
+## used for temporarily caching failures (e.g., 404's for 10 seconds)
+##
+## By default, each Origin Configuration maps to the 'default' negative cache which you can
+## configure below, or can define your own negative caches, and specify them in your origin configs.
+## See /docs/negative-caching.md for more info.
+##
+
+# [negative_caches]
+# [negative_caches.default]
+# # The 'default' negative cache config, mapped by all origins by default,
+# # is empty unless you populate it. Update it by adding entries here in the format of:
+# # code = ttl_secs
+
+## Here's a pre-populated negative cache config ready to be uncommented and used in an origin config
+## The 'general' negative cache config will cache common failure response codes for 3 seconds
+# [negative_caches.general]
+# 400 = 3
+# 404 = 3
+# 500 = 3
+# 502 = 3
+
+# Configuration options for mapping Origin(s)
+[origins]
+
+ # example origin named default. default is always created with these settings unless a different origin is defined here.
+ [origins.default]
+
+ ## is_default describes whether this origin is the default origin considered when routing http requests
+ ## it is false, by default; but if you only have a single origin configured, is_default will be true unless explicitly set to false
+ # is_default = true
+
+ # origin_type identifies the origin type.
+ # Valid options are: 'prometheus', 'influxdb', 'clickhouse', 'irondb', 'reverseproxycache' (or just 'rpc')
+ # origin_type is a required configuration value
+ origin_type = 'prometheus'
+
+ ## cache_name identifies the name of the cache (configured above) that you want to use with this origin proxy. default is 'default'
+ # cache_name = 'default'
+
+ ## cache_key_prefix defines the prefix this origin appends to cache keys. When using a shared cache like Redis,
+ ## this can help partition multiple trickster instances that may have the same same hostname or ip address (the default prefix)
+ # cache_key_prefix = 'example'
+
+ ## negative_cache_name identifies the name of the negative cache (configured above) to be used with this origin. default is 'default'
+ # negative_cache_name = 'default'
+
+ # origin_url provides the base upstream URL for all proxied requests to this origin.
+ # it can be as simple as http://example.com or as complex as https://example.com:8443/path/prefix
+ # origin_url is a required configuration value
+ origin_url = 'http://prometheus:9090'
+
+ ## dearticulate_upstream_ranges, when true, instructs Trickster to make multiple parallel requests to the origin for each
+ ## range needed to fulfill the client request, rather than making a multipart range request. default is false
+ # dearticulate_upstream_ranges = false
+
+ ## multipart_ranges_disabled, when true, instructs Trickster to return the full object when the client provides
+ ## a multipart range request. The default is false.
+ # multipart_ranges_disabled = false
+
+ ## compressable_types defines the Content Types that will be compressed when stored in the Trickster cache
+ ## reasonable defaults are set, so use this with care. To disable compression, set compressable_types = []
+ ## Default list is provided here:
+ # compressable_types = [ 'text/javascript', 'text/css', 'text/plain', 'text/xml', 'text/json', 'application/json', 'application/javascript', 'application/xml' ]
+
+ ## timeout_secs defines how many seconds Trickster will wait before aborting and upstream http request. Default: 180s
+ # timeout_secs = 180
+
+ ## keep_alive_timeout_secs defines how long Trickster will wait before closing a keep-alive connection due to inactivity
+ ## if the origin's keep-alive timeout is shorter than Trickster's, the connect will be closed sooner. Default: 300
+ # keep_alive_timeout_secs = 300
+
+ ## max_idle_conns set the maximum concurrent keep-alive connections Trickster may have opened to this origin
+ ## additional requests will be queued. Default: 20
+ # max_idle_conns = 20
+
+ ## backfill_tolerance_secs prevents new datapoints that fall within the tolerance window (relative to time.Now) from being cached
+ ## Think of it as "never cache the newest N seconds of real-time data, because it may be preliminary and subject to updates"
+ ## default is 0
+ # backfill_tolerance_secs = 0
+
+ ## timeseries_retention_factor defines the maximum number of recent timestamps to cache for a given query. Default is 1024
+ # timeseries_retention_factor = 1024
+
+ ## timeseries_ttl_secs defines the relative expiration of cached timeseries. default is 6 hours (21600 seconds)
+ # timeseries_ttl_secs = 21600
+
+ ## timeseries_eviction_method selects the metholodogy used to determine which timestamps are removed once
+ ## the timeseries_retention_factor limit is reached. options are 'oldest' and 'lru'. Default is 'oldest'
+ # timeseries_eviction_method = 'oldest'
+
+ ## fast_forward_disable, when set to true, will turn off the 'fast forward' feature for any requests proxied to this origin
+ # fast_forward_disable = false
+
+ ## fastforward_ttl_secs defines the relative expiration of cached fast forward data. default is 15s
+ # fastforward_ttl_secs = 15
+
+ ## max_ttl_secs defines the maximum allowed TTL for any object cached for this origin. default is 86400
+ # max_ttl_secs = 86400
+
+ ## revalidation_factor is the multiplier for object lifetime expiration to determine cache object TTL; default is 2
+ ## for example, if a revalidatable object has Cache-Control: max-age=300, we will cache for 10 minutes (300s * 2)
+ ## so there is an opportunity to revalidate
+ # revalidation_factor = 2
+
+ ## max_object_size_bytes defines the largest byte size an object may be before it is uncacheable due to size. default is 524288 (512k)
+ # max_object_size_bytes = 524288
+
+ ##
+ ## Each origin type implements their own defaults for health_check_upstream_url, health_check_verb and health_check_query,
+ ## which can be overridden per origin. See /docs/health.md for more information
+
+ ## health_check_upstream_url is the URL Trickster will request against this origin to
+ ## when a health check request is received by Trickster via http://
+
+## Progressive Collapsed Forwarding
+
+Progressive Collapsed Forwarding (PCF) is an improvement upon the basic version, in that it eliminates the waitlist and serves all simultaneous requests concurrently while the object is still downloading from the server, similar to Apache Traffic Server's "read-while-write" feature. This may be useful in low-latency applications such as DASH or HLS video delivery, since PCF minimizes Time to First Byte latency for extremely popular objects.
+
+The feature is further detailed in the following diagram:
+
+
+
+### PCF for Proxy-Only Requests
+
+Trickster provides a unique feature that implements PCF in Proxy-Only configurations, to bring the benefits of Collapsed Forwarding to HTTP Paths that are not configured to be routed through the Reverse Proxy Cache. (See [Paths](./paths.md) documentation for more info on routing).
+
+The feature is further detailed in the following diagram:
+
+
+
+## How to enable Progressive Collapsed Forwarding
+
+When configuring path configs as described in [Paths Documentation](./paths.md) you simply need to add `progressive_collapsed_forwarding = true` in any path config using the `proxy` or `proxycache` handlers.
+
+Example:
+
+```toml
+ [origins.test.paths]
+ [origins.test.paths.thing1]
+ path = '/test_path1/'
+ match_type = 'prefix'
+ handler = 'proxycache'
+ progressive_collapsed_forwarding = true
+
+ [origins.test.paths.thing2]
+ path = '/test_path2/'
+ match_type = 'prefix'
+ handler = 'proxy'
+ progressive_collapsed_forwarding = true
+```
+
+See the [example.conf](../cmd/trickster/conf/example.conf) for more configuration examples.
+
+## How to test Progressive Collapsed Forwarding
+
+An easy way to test PCF is to set up your favorite file server to host a large file(Lighttpd, Nginx, Apache WS, etc.), In Trickster turn on PCF for that path config and try make simultaneous requests.
+If the networking between your machine and Trickster has enough bandwidth you should see both streaming at the equivalent rate as the origin request.
+
+Example:
+
+- Run a Lighttpd instance or docker container on your local machine and make a large file available to be served
+- Run Trickster locally
+- Make multiple curl requests of the same object
+
+You should see the speed limited on the origin request by your disk IO, and your speed between Trickster limited by Memory/CPU
diff --git a/docs/configuring.md b/docs/configuring.md
index 51f951a21..11e2e38b6 100644
--- a/docs/configuring.md
+++ b/docs/configuring.md
@@ -6,9 +6,11 @@ There are 3 ways to configure Trickster, listed here in the order of evaluation.
* Environment Variables
* Command Line Arguments
+Note that while the Confifguration file provides a very robust number of knobs you can adjust, the ENV and CLI Args options support only basic use cases.
+
## Internal Defaults
-Internal Defaults are set for all configuration values, and are overridden by the configuration methods described below. All Internal Defaults are described in [conf/example.conf](../conf/example.conf) comments.
+Internal Defaults are set for all configuration values, and are overridden by the configuration methods described below. All Internal Defaults are described in [cmd/trickster/conf/example.conf](../cmd/trickster/conf/example.conf) comments.
## Configuration File
@@ -16,16 +18,17 @@ Trickster accepts a `-config /path/to/trickster.conf` command line argument to s
When a `-config` parameter is not provided, Trickster will check for the presence of a config file at `/etc/trickster/trickster.conf` and load it if present, or proceed with the Internal Defaults if not present.
-Refer to [conf/example.conf](../conf/example.conf) for full documentation on format of a configuration file.
+Refer to [cmd/trickster/conf/example.conf](../cmd/trickster/conf/example.conf) for full documentation on format of a configuration file.
## Environment Variables
Trickster will then check for and evaluate the following Environment Variables:
-* `TRK_ORIGIN=http://prometheus.example.com:9090` - The default origin to proxy Prometheus requests
+* `TRK_ORIGIN=http://prometheus.example.com:9090` - The default origin for proxying all http requests
+* `TRK_ORIGIN_TYPE=prometheus` - The type of [supported origin server](./supported-origin-types.md)
* `TRK_LOG_LEVEL=INFO` - Level of Logging that Trickster will output
* `TRK_PROXY_PORT=8000` -Listener port for the HTTP Proxy Endpoint
-* `TRK_METRICS_PORT=8001` - Listener port for the HTTP Metrics Endpoint
+* `TRK_METRICS_PORT=8001` - Listener port for the Metrics and pprof debugging HTTP Endpoint
## Command Line Arguments
@@ -33,6 +36,7 @@ Finally, Trickster will check for and evaluate the following Command Line Argume
* `-log-level INFO` - Level of Logging that Trickster will output
* `-config /path/to/trickster.conf` - See [Configuration File](#configuration-file) section above
-* `-origin http://prometheus.example.com:9090` - The default origin to proxy Prometheus requests
+* `-origin http://prometheus.example.com:9090` - The default origin for proxying all http requests
+* `-origin-type prometheus` - The type of [supported origin server](./supported-origin-types.md)
* `-proxy-port 8000` - Listener port for the HTTP Proxy Endpoint
-* `-metrics-port 8001` - Listener port for the HTTP Metrics Endpoint
+* `-metrics-port 8001` - Listener port for the Metrics and pprof debugging HTTP Endpoint
diff --git a/docs/developer/adding-new-config.md b/docs/developer/adding-new-config.md
new file mode 100644
index 000000000..8f90dd4c1
--- /dev/null
+++ b/docs/developer/adding-new-config.md
@@ -0,0 +1,34 @@
+# Adding a New Configuration Value
+
+Trickster configurations are defined in `./internal/config/` and are mapped to `toml` annotations.
+
+When adding a configuration value, there are several places to add references, which are described below.
+
+## Configuration Code
+
+Each new configuration value must be defined in the `config` package under an existing Configuration collection (Origins, Caches, Paths, etc.).
+
+Make sure the TOML annotation uses a `lowercase_no_spaces` naming convention, while the configuration member name itself should be `CamelCase`. Follow the existing configs for guidance.
+
+Once you have defined the configuration member, if it is part of a `CacheConfig`, `OriginConfig` or `PathConfig`, it must also be added to the configuration parser for the specific type of configuration. These methods iterate through known TOML annoations to survey which configs have been set. This allows Trickster to know if a value is set because it is the initialized default value or because the operator has explicitly set it.
+
+## Feature Code
+
+Once you have defined your configuration value(s), you must put them to work by referencing them elsewhere in the Trickster code, and used to determine or customize the application functionality. Exactly where this happens in the code depends upon the context and reach or your new configuration, and what features its state affects. Consult with a project maintainer if you have any questions.
+
+## Tests
+
+All new values that you add should have accompanying unit tests to ensure the modifications the value makes to the application in the feature code work as designed. Unit Tests should include verification of: proper parsing of configuration value from test config files (in ./testdata), correct feature functionality enable/disable based on the configuration value, correct feature implementation, coverage of all executable lines of code. Unit Test will span the `config` package, and any package(s) wherein the configuration value is used by the applciation.
+
+## Documentation
+
+The feature should be documented under `./docs` directory, in a suitable existing or new markdown file based on the nature of the feature. The documentation should show the key example configuration options and describe their expected results, and point to the example config file for more information.
+
+The example config file (./cmd/trickster/conf/example.conf) should be updated to include the exhaustive description and options for the configuration value(s).
+
+## Deployment
+
+The `./deply/kube/configmap.yaml` must be updated to include the new configuration option(s). Generally this file contains a copy/paste of `./cmd/trickster/conf/example.conf`.
+
+The `./deploy/helm/trickster/values.yaml` file must be updated to mirror the configuration option(s) in the example.conf, and `./deploy/helm/trickster/templates/configmap.yaml` must be updated to map any new `yamlCaseValues` to their respective `toml_style_values` for config file generation via the template.
+
diff --git a/docs/developer/origin-extensibility.md b/docs/developer/origin-extensibility.md
new file mode 100644
index 000000000..7c3cc2203
--- /dev/null
+++ b/docs/developer/origin-extensibility.md
@@ -0,0 +1,116 @@
+# Extending Trickster to Support a New Origin Type
+
+Trickster 1.0 was written with extensibility in mind, and should be able to work with any time series database that has an HTTP-based API. In Trickster, we generically refer to our supported TSDB's as Origin Types. Some Origin Types are easier to implement and maintain than others, depending upon a host of factors that are covered later in this document. This document is meant to help anyone wishing to extend Trickster to support a new Origin Type, particularly in gauging the level of effort, understanding what is involved, and implementing the required interfaces and rules.
+
+## Qualifications
+
+Not every database server out there is a candidate for being fronted by Trickster. Trickster serves the specific purpose of accelerating the delivery of time series data sets, and will not benefit traditional relational databases, NoSQL, etc.
+
+As mentioned, the database must be able to be queried for and return time series data via HTTP. Some databases that are not specifically TSDB's actually do support querying for and returning data in a time series format, and Trickster will support those cases as detailed below, so long as they have an HTTP API.
+
+### Skills Needed
+
+In addition to these requirements in the technology, there are also skills qualifications to consider.
+
+Whether or not you've contributed to Open Source Software before, take a look at our Contributing guidelines so you know how the process works for the Trickster project. If you are unfamiliar with the Forking Workflow, read up on it so that you are able to contribute to the project through Pull Requests.
+
+Trickster is a 100% Go project, so you will need to have experience writing in Go, and in particular, data marshaling/unmarshaling and data set manipulation (sorting, merging, cropping, de-duplicating, etc.). You will need to have a good understanding of the prospective Origin Type's query language and response payload structure, so you can write the necessary parsing and modeling methods that allow Trickster to manipulate upstream HTTP requests and merge newly fetched data sets into the cache.
+
+While this might sound daunting, it is actually much easier than it appears on the surface. Since Trickster's DeltaProxyCache engine does a lot of the heavy lifting, you only have to write a series of interface functions before finding yourself near the finish line. And since a few Origin Types are already implemented, you can use their implementations for references, since the logic for your prospective Origin Type should be similar.
+
+## Interfaces
+
+Trickster provides 2 required interfaces for enabling a new Origin Type: The Proxy Client and the Time Series
+
+### Proxy Client Interface
+
+The Proxy Client Interface ([code](https://github.com/Comcast/trickster/blob/next/internal/proxy/model/client.go)) is used by Trickster to manipulate HTTP requests and responses in order to accelerate the requests.
+
+For your Proxy Client Implementation, you will need to know these things about the Origin:
+
+- What URL paths and methods must be supported, and which engine through which to route each path (Basic HTTP Proxy, Object Proxy Cache, or Time Series Delta Proxy Cache). The proxy engines will call your client implementation's interface exports in order to service user requests.
+
+- What data inputs the origin expects (Path, URL parameters, POST Data, HTTP Headers, cookies, etc.), and how to manipulate the query's time range when constructing those inputs to achieve a desired result.
+
+The Proxy Client Interface Methods you will need to implement are broken into several groups of functionality, as follows.
+
+#### Basic Getters
+
+- `Configuration` returns the \*config.OriginConfig object for the origin.
+
+- `Name` returns the configured name of the Origin Type instance.
+
+- `HTTPClient` returns the reusable \*http.Client object that communicates with the Origin.
+
+#### HTTP Request Routing and Handling
+
+- `RegisterRoutes` registers all of the HTTP Paths that will be used by the Origin Type and map them to handlers written to service the various paths.
+
+- `HealthHandler` this method is a standard HTTP Handler that can verify and report back the health of the upstream origin and the proxy's connection to it. You will certainly create at least one other handler in your Origin Type package, but this is the only one required for conformance to the Proxy Client interface.
+
+#### Caching
+
+- `DeriveCacheKey` inspects the client request and returns the corresponding cache key
+
+#### Time Series Handling
+
+- `UnmarshalTimeseries` deserializes an HTTP Response time series payload into a Go struct
+
+- `MarshalTimeseries` seralizes a time-series struct into a serialized byte slice.
+
+- `UnmarshalInstantaneous` deserializes an HTTP Response instantaneous payload into a Go struct. This may not be applicable to every potential Origin Type.
+
+- `ParseTimeRangeQuery` inspects the client request and returns a corresponding timeseries.TimeRangeQuery
+
+- `SetExtent` updates an upstream request's time range parameters as needed based on the delta gap analysis
+
+- `FastForwardURL` returns the URL to the origin to collect Fast Forward data points based on the provided HTTP Request
+
+### Time Series Interface
+
+The Time Series Interface ([code](https://github.com/Comcast/trickster/blob/next/internal/timeseries/timeseries.go)) is used by Trickster to manipulate Time Series documents in order to maintain the cache and construct downstream client request bodies.
+
+For your Time Series Implementation, you will need to know these things about the Origin:
+
+- The structure of the response payload and how that translates into Go structs. More often than not, a prospective Origin Type offers an importable model to assist with this.
+
+The Time Series Interface Methods you will need to implement are broken into several groups of functionality, as follows.
+
+#### Getters
+
+- `Extents` returns a list of the time ranges present in the cache
+- `Step` returns the Step (the duration between each timestamp in the series)
+- `SeriesCount` returns the number of series (e.g., graph lines) in the data set
+- `ValueCount` returns the total number of values across all series in the data set
+
+#### Setters
+
+- `SetExtents` sets the list of the time ranges present in the cache
+- `SetStep` sets the Step
+
+#### Data Set Manipulation
+
+- `Merge` merges a variadic list of time series into the base time series
+- `Sort` chronologically sorts the values in each series in the time series
+- `Copy` makes an new exact copy of the time series
+- `Crop` removes any values from the time series that are outside of the provided time range
+
+## Special Considerations
+
+### Query Language Complexity
+
+One of the main areas of consideration is the complexity of parsing and manipulating an inbound query. You will need to (1) determine if it is indeed a request for a timeseries; and if so (2) extract the requested time range and step duration for the query; and in the event of a partial cache hit, (3) adjust the time range for the query to a provided range - all of which allows the DeltaProxyCache to fetch just the needed sections of data from the upstream origin. Requirements 1 and 2 are functionality in `ParseTimeRangeQuery` while requirement 3 is the functionality of `SetExtent`. The overall complexity of this process can significantly affect the level of effort required to implement a new Origin Type.
+
+In the example of Prometheus, the process was extremely simple: since, in the Prometheus HTTP API, time range queries have a separate http endpoint path from instantaneous queries, and because the time range is provided as separate query parameters from the query itself, the range is easily modified without Trickster having any knowledge of the underlying query or having to even parse it at all.
+
+In the example of ClickHouse, the process is much harder: since the query language is a variant of SQL standard, the requested time ranges are embedded into the query itself behind the `WHERE` clause. In cases such as this, Trickster must have some way, either through (1) importing a database package that can deserialize the query, allow manipulation, and serialize the modified query for you; or (2) new parsing and search/replacement logic introduced in your own package - which allows the Client to interepret the time range and modify it with time ranges provided by the DeltaProxyCache. With ClickHouse, since it is a C++ project and Trickster is Golang, we could not import any package to handle this work, so we crafted a regular expression to match against inbound queries. This regex extracts the timerange (and any ClickHouse time modifiers like `startOfMinute`) and step value as well as any other areas that include the timerange, and then use simple builtin string functions to inject tokens in place of specific ranges in the provided query. Then, when SetExtent is called, those tokens are search/replaced with the provided time values.
+
+### Data Model Considerations
+
+Once you have the Client Interface implementation down and can interact with the upstream HTTP API, you will turn your attention to managing the response payload data and what to do with it, and that happens in your Timeseries Interface implementation. And like the Client interface, it will come with its own unique challenges.
+
+The main consideration here is the format of the output and what challenges are presented by it. For example, does the payload include any required metadata (e.g., a count of total rows returned) that you will need to synthesize within your Timeseries after a `Merge`, etc. Going back to the ClickHouse example, since it is a columnar database that happens to have time aggregation functions, there are a million ways to formulate a query that yields time series results. That can have implications on the resulting dataset: which fields are the time and value fields, and what are the rest? Are all datapoints for all the series in a single large slice or have they been segregated into their own slices? Is the Timestamp in Epoch format, and if so, does it represent seconds or milliseconds? In order to support an upstream database, you may need to establish or adopt guidelines around these and other questions to ensure full compatibility. The ClickHouse plugin for Grafana requires that for each datapoint of the response, the first field is the timestamp and the second field is the numeric value - so we adopt and document the same guideline to conform to existing norms.
+
+## Getting More Help
+
+On the Gophers Slack instance, you can find us on the #trickster channel for any help you may need.
diff --git a/docs/health.md b/docs/health.md
index 5cecd5b37..c86ac0510 100644
--- a/docs/health.md
+++ b/docs/health.md
@@ -1,15 +1,23 @@
# Health Checks
-## Ping Endpoint
-Trickster provides a `/ping` endpoint that returns a response of `200 OK` and the word `pong` if Trickster is up and running. The `/ping` endpoint does not check any proxy configurations or upstream origins.
+## Trickster Service Health - Ping Endpoint
-## Health Check Endpoint
-Trickster offers a `/health` endpoint for monitoring the health of the Trickster service and its upstream connection to the origin. To test the upstream origin, Trickster will make a request to its labels endpoint (`/label/__name__/values`).
+Trickster provides a `/trickster/ping` endpoint that returns a response of `200 OK` and the word `pong` if Trickster is up and running. The `/trickster/ping` endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.
-An HTTP response of `200 OK` indicates that the end-to-end request to the origin was successful.
+## Upstream Connection Health - Origin Health Endpoints
-In a multi-origin setup, requesting against `/health` will test the default origin. You can indicate a specific origin to test by crafting requests in the same way a normal multi-origin request is structured. For example, `/origin_moniker/health`. See [multi-origin.md](multi-origin.md) for more information.
+Trickster offers `health` endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.
+
+Each configured origin's health check path is `/trickster/health/ORIGIN_NAME`. For example, if your origin is named `foo`, you can perform a health check of the upstream URL at `http://
-#### Using urlparam as the Multi-Origin Indicator
-
-The client provides the origin moniker as an url param.
-
-Example Client Request URLs:
-
-* To Request from Origin `foo`: http://trickster.example.com:9090/query?origin=foo&query=xxx
-
-* To Request from Origin `bar`: http://trickster.example.com:9090/query?origin=bar&query=xxx
-
### DNS Alias Configuration
-In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster represents the Origin Moniker. Therefore, the entire FQDN must be part of the configuration section header. In this mode, the path and url params requested against Trickster are _not_ modified to specify an origin.
+In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster represents the Origin Name. Therefore, the entire FQDN must be part of the configuration section header. In this mode, the URL Path is _not_ considered during Origin Selection.
Example DNS-based Origin Configuration:
```
[origins]
- # default origin
- [origins.default]
+ # origin1 origin
+ [origins.origin1]
origin_url = 'http://prometheus.example.com:9090'
- api_path = '/api/v1'
- default_step = 300
- ignore_no_cache_header = false
- max_value_age_secs = 86400
+ origin_type = 'prometheus'
+ cache_name = 'default'
+ is_default = true
# "foo" origin
- [origins.trickster-foo.example.com]
+ [origins.'trickster-foo.example.com']
origin_url = 'http://prometheus-foo.example.com:9090'
- api_path = '/api/v1'
- default_step = 300
- ignore_no_cache_header = false
- max_value_age_secs = 86400
+ origin_type = 'prometheus'
+ cache_name = 'default'
# "bar" origin
- [origins.trickster-bar.example.com]
+ [origins.'trickster-bar.example.com']
origin_url = 'http://prometheus-bar.example.com:9090'
- api_path = '/api/v1'
- default_step = 300
- ignore_no_cache_header = false
- max_value_age_secs = 86400
+ origin_type = 'prometheus'
+ cache_name = 'default'
```
Example Client Request URLs:
-* To Request from Origin `foo`: http://trickster-foo.example.com:9090/query?query=xxx
-* To Request from Origin `bar`: http://trickster-bar.example.com:9090/query?query=xxx
+* To Request from Origin `foo`: 
+
-Single "Everything" is the most common placement model. In this configuration, you have one dashboard endpoint, one Trickster endpoint and one Prometheus endpoint. Behind each endpoint, you may have a single instance or a cluster. Each component is only aware of the other component's endpoint exposure and not the underlying configuration. This configuration represents a one-for-one-for-one deployment of your Dashboard, Prometheus, and Trickster endpoints.
+Single "Everything" is the most common placement model. In this configuration, you have one optional dashboard endpoint, one Trickster endpoint and one HTTP or TSDB endpoint. Behind each endpoint, you may have a single instance or a cluster. Each component is only aware of the other component's endpoint exposure and not the underlying configuration. This configuration represents a one-for-one-for-one deployment of your Dashboard, Origin, and Trickster endpoints.
-## Multi-Origin
+## Multiple Origins
-
+
-In a Multi-Origin placement, you have one dashboard endpoint, one Trickster endpoint, and multiple Prometheus endpoints. Trickster is aware of each Prometheus endpoint and treats them as unique databases to which it proxies and caches data independently of each other.
+In a Multi-Origin placement, you have one dashboard endpoint, one Trickster endpoint, and multiple TSDB and/or HTTP endpoints. Trickster is aware of each upstream endpoint and treats each as a unique origin to which it proxies and caches data independently from the others. Trickster selects the origin based on Host Header or URL Path from the client request.
-This is a good configuration to use when you have a single dashboard that displays data about multiple redundant clusters (each with its own Prometheus), or when you have a single dashboard representing information about many different kinds of systems. For example, if you operate a "Dashboard as a Service" solution under which many teams use your Dashboard system by designing their own dashboard screens and bringing their own databases, a single Trickster endpoint can be used to accelerate dashboards for all of your customers.
+This setup may benefit situations where you have one ore more a static file server origins serving HTML, CSS and JavaScript assets and/or one or more API endpoints, all supporting a common platform.
-You will need to configure each Trickster-to-Prometheus mapping separately in your dashboard application as a separately named Prometheus data source. Refer to the [multi-origin](./multi-origin.md) documentation for configuring multi-origin support in Trickster and Grafana.
+For Time Series Dasbhoard acceleration, this is a good configuration to use when you have a single dashboard that displays data about multiple redundant clusters (each with its own TSDB), or when you have a single dashboard representing information about many different kinds of systems. For example, if you operate a "Dashboard as a Service" solution under which many teams use your Dashboard system by designing their own dashboard screens and bringing their own databases, a single Trickster endpoint can be used to accelerate dashboards for all of your customers.
+
+You will need to configure each Trickster-to-TSDB mapping separately in your dashboard application as a separately named TSDB data source. Refer to the [multi-origin](./multi-origin.md) documentation for configuring multi-origin support in Trickster and Grafana.
In this configuration, be aware that the default 'memory' cache may be underpowered depending on the number of customers, as well as the size and number of queries that need to be cached by each customer. Refer to the [caches](./caches.md) document to select and configure the caching layers as needed to meet your specific situation.
## Multi-Trickster
-
+
-In a Multi-Trickster configuration, you have one dashboard endpoint, multiple Trickster endpoints, and multiple Prometheus endpoints, with each Trickster Endpoint having a one-to-one mapping to a Prometheus Endpoint as a pair. This is a good design if Multi-Origin is not performant enough for the amount of activity associated with your solution (e.g., you need more Tricksters). If the Dashboard system owner is different from the Prometheus system owner, either party could own and operate the Trickster instance.
+In a Multi-Trickster configuration, you have one dashboard endpoint, multiple Trickster endpoints, and multiple TSDB or HTTP endpoints, with each Trickster Endpoint having a one-to-one mapping to a TSDB/HTTP Endpoint as a pair. This is a good design if Multi-Origin is not performant enough for the amount of activity associated with your solution (e.g., you need more Tricksters). If the Dashboard system owner is different from the TSDB system owner, either party could own and operate the Trickster instance.
diff --git a/docs/promsim.md b/docs/promsim.md
new file mode 100644
index 000000000..d5859032e
--- /dev/null
+++ b/docs/promsim.md
@@ -0,0 +1,102 @@
+# PromSim - a barebones Prometheus data simulator
+
+PromSim is a golang package available at `github.com/Comcast/trickster/pkg/promsim` that facilitates unit testing of components that are direct consumers of Prometheus JSON data. It works by simulating datasets, output in the Prometheus's v1 HTTP API format, that consist of values that are repeatably generatable for the provided query and timerange inputs. The data output by PromSim does not represent reality in any way, and is only useful for unit testing and integration testing, by providing a synthesized Prometheus environment that outputs meaningless data. None of PromSim's result sets are stored on or retrieved from disk, and are calculated just-in-time on every request, using simple mathematical computations. In Trickster, we use PromSim to conduct end-to-end testing of our DeltaProxyCache during unit testing, without requiring a real Prometheus server.
+
+## Supported Simulation Endpoints
+
+- `/query` (Instantaneous)
+- `/query_range` (Time Series)
+
+## Example Usage in Unit Testing
+
+PromSim only uses builtin golang packages and should thus work out-of-the-box without any other dependency concerns.
+
+```go
+package mypackage
+
+import (
+ "io/ioutil"
+ "net/http"
+ "testing"
+
+ "github.com/Comcast/trickster/pkg/promsim"
+)
+
+func TestPromSim(t *testing.T) {
+
+ ts := promsim.NewTestServer()
+ client := &http.Client{}
+ const expected = `{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"random_label":"57","series_count":"1","series_id":"0"},"values":[[2,"93"]]}]}}`
+ resp, err := client.Get(ts.URL + `/api/v1/query_range?query=my_test_query{random_label="57",series_count="1"}&start=2&end=2&step=15`)
+ if err != nil {
+ t.Error(err)
+ }
+
+ body, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(body) != expected {
+ t.Errorf("expected [%s] got [%s]", expected, string(body))
+ }
+}
+```
+
+## Behavior Modifiers
+
+PromSim's behavior can be modified in several ways, on a per-query basis, to produce a desired behavior. This is done by providing specific query label values as part of your test queries. All modifier labels are optional, and can be used together in any possible combination without conflict. Providing the same modifier label more than once in a query will result in the last instance of the modifier to be used when constructing the response values.
+
+### Series Count
+
+By default, PromSim will only return a single series in the result set. You can provide a label of `series_count` to indicate the exact number of series that should be returned.
+
+Example query that returns 3 series: `query=my_test_query{series_count="3"}&start=2&end=2&step=15`
+
+### Line Pattern
+
+By default, PromSim uses a "repeatable number generator" to output data. Under the hood, it works by re-seeding `math.Rand` with a hashed value for the provided query string and the timestamp for which a value is needed, and returning the first value from the generator after seeding.
+
+You can provide a `line_pattern` label to utilize other supported number generators. The options are `repeatable_random` (default, described above) and `usage_curve`.
+
+`usage_curve` will return numbers that follow a simulated usage curve pattern (rising in the afternoon, peaking in the evening, troughing overnight).
+
+Example using the usage_curve line pattern: `query=my_test_query{series_count="3",max_value="250",min_value="10",line_pattern="usage_curve"}&start=2&end=2&step=15`
+
+### Latency
+
+PromSim is capable of simulating latency by accepting 2 optional query labels: `latency_ms` and `range_latency_ms`. Both labels can be used in conjunction to produce a desired effect.
+
+#### Upfront Latency
+
+The `latency_ms` label introduces an upfront static processing latency of the provided duration on each http response. This is useful in simulating roundtrip wire latency.
+
+Example adding 300ms of upfront latency: `query=my_test_query{latency_ms="300"}&start=2&end=2&step=15`
+
+#### Range Latency
+
+The `range_latency_ms` label produces a per-unique-value latency effect. The result is that the response from PromSim will be delayed by a certain amount, depending upon on the number of series, size of desired timerange and step value. This is useful in simulating very broad label scopes that slow down query response times in the real world.
+
+Example adding 5ms of range latency: `query=my_test_query{range_latency_ms="5",series_count="2"}&start=0&end=1800&step=15`. In this example, 1.2s of total latency is introduced (120 datapoints x 2 series x 5ms) into the HTTP response.
+
+### Min and Max Values
+
+The `min_value` and `max_value` labels allow you to define the extent of possible values returned by PromSim in the result set, and are fairly straightforward. The default min and max values, when not customized, are 0 and 100, respectively.
+
+Example of min and max: `query=my_test_query{series_count="2",min_value="32",max_value="212"}&start=0&end=90&step=15`. In this case, the returned values will be between 32 and 212, rather than 0 and 100.
+
+### Status Code
+
+The `status_code` label will cause PromSim to return the provided status code instead of `200 OK`. This is useful for testing simulated failcases such as invalid query parameters.
+
+Example query that returns 400 Bad Request: `query=my_test_query{status_code="400"}&start=2&end=2&step=15`
+
+### Invalid Response Body
+
+The `invalid_response_body` label, when provided and set to a value other than 0, will cause PromSim to return a response that cannot be deserialized into a Prometheus Matrix or Vector object, which is again useful for testing failure handling within your app.
+
+Example query that returns invalid response: `query=my_test_query{invalid_response_body="1"}&start=2&end=2&step=15`
+
+## Example Usage as Standalone App
+
+Trickster provides a sample standalone implementation of PromSim. This is useful for backing full simulation dashboards or running a local background app for querying during development of a Prometheus consumer app. You can find it at `github.com/Comcast/trickster/cmd/promsim`, and, from that working directory, simply run `go run *.go \[PORT\]`. If a port number is not provided, it defaults to 9090, just like Prometheus.
diff --git a/docs/range_request.md b/docs/range_request.md
new file mode 100644
index 000000000..174e0a106
--- /dev/null
+++ b/docs/range_request.md
@@ -0,0 +1,19 @@
+# Range Request Support
+
+Trickster's HTTP Reverse Proxy Cache offers best-in-class acceleration and caching of Range Requests.
+
+Much like its Time Series Delta Proxy Cache, Trickster's Reverse Proxy Cache will determine what ranges are cached, and only request from the origin any uncached ranges needed to service the client request, reconstituting the ranges within the cache object. This ensures minimal response time in the event of a cache miss.
+
+In addition to supporting basic single-Range requests (`Range: bytes=0-5`) Trickster also supports Multipart Range Requests (`Range: bytes=0-5, 10-20`).
+
+In the event that an upstream does not support Multipart Range Requests, Trickster offers a unique feature called Upstream Range Dearticulation, that will separate any ranges needed from the origin into individual, parallel HTTP requests, which are reconstituted by Trickster. This feature can be enabled for an origin by setting `dearticulate_upstream_ranges = true`, as in this example:
+
+```toml
+[origins]
+ [origins.default]
+ origin_type = 'reverseproxycache'
+ origin_url = 'http://example.com/'
+ dearticulate_upstream_ranges = true
+```
+
+In the event that downstream clients should not expect MultiPart Range Request support, Trickster offers a setting to fully disable support on a per-origin basis. Set `multipart_ranges_disabled = true` and Trickster will strip Range Request headers that include multiple Ranges, which will result in a 200 OK response with the full body.
diff --git a/docs/retention.md b/docs/retention.md
new file mode 100644
index 000000000..7ccc5bdd9
--- /dev/null
+++ b/docs/retention.md
@@ -0,0 +1,45 @@
+# Trickster Caching Retention Policies
+
+## Basic HTTP Origins
+
+Trickster will respect HTTP 1.0, 1.1 and 2.0 caching directives from both the downstream client and the upstream origin when determining objectly cacheability and TTL. You can override the TTL by setting a custom `Cache-Control` header on a per-[Path Config](./paths.md) basis.
+
+### Cache Object Evictions
+
+If you use a Trickster-managed cache (Memory, Filesystem, bbolt), then a maximum cache size is maintained by Trickster. You can configure the maximum size in number of bytes, number of objects, or both. See the example configuration for more information.
+
+Once the cache has reached its configured maximum size of objects or bytes, Trickster will undergo an eviction routine that removes cache objects until the size has fallen below the configured maximums. Trickster-managed caches maintain a last access time for each cache object, and utilizes a Least Recently Used (LRU) methodology when selecting objects for eviction.
+
+Caches whose object lifetimes are not managed internally by Trickster (Redis, BadgerDB) will use their own policies and methodologies for evicting cache records.
+
+## Time Series Origins
+
+For non-time series responses from a TSDB, Trickster will adhere to HTTP caching rules as directed by the downstream client and upstream origin.
+
+For time series data responses, Trickster will cache as follows:
+
+### TTL Settings
+
+TTL settings for each Origin configured in Trickster can be customized independently of each other, and separate TTL configurations are available for timeseries objects, and fast forward data. See [cmd/trickster/conf/example.conf](../cmd/trickster/conf/example.conf) for more info on configuring default TTLs.
+
+### Time Series Data Retention
+
+Separately from the TTL of a time series cache object, Trickster allows you to control the size of each timeseries object, represented as a count of maximum timestamps in the cache object, on a _per origin_ basis. This configuration is known as the `timeseries_retention_factor` (TRF), and has a default of 1024. Most dashboards for most users request and display approximately 300-to-400 timestamps, so the default TRF allows users to still recall recently-displayed data from the Trickster cache for a period of time after the data has aged off of real-time views.
+
+If you have users with a high-resolution dashboard configuration (e.g., a 24-hour view with a 1-minute step, amounting to 1440 data points per graph), then you may benefit from increasing the `timeseries_retention_factor` accordingly. If you use a managed cache (see [caches](./caches.md)) and increase the `timeseries_retention_factor`, the overall size of your cache will not change; the result will be fewer objects in cache, with the timeseries objects having a larger share of the overall cache size with more aged data.
+
+#### Time Series Data Evictions
+
+Once the TRF is reached for a time series cache object, Trickster will undergo a timestamp eviction process for the record in question. Unlike the Cache Object Eviction, which removes an object from cache completely, TRF evictions examine the data set contained in a cache object and remove timestamped data in order to reduce the object size down to the TRF.
+
+Time Series Data Evictions apply to all cached time seres data sets, regardless of whether or not the cache object lifecycle is managed by Trickster.
+
+Trickster provides two eviction methodologies (`timeseries_eviction_method`) for time series data eviction: `oldest` (default) and `lru`, and is configurable per-origin.
+
+When `timeseries_eviction_method` is set to `oldest`, Trickster maintains time series data by calculating the "oldest cacheable timestamp" value upon each request, using `time.Now().Add(step * timeseries_retention_factor * -1)`. Any queries for data older than the oldest cacheable timestamp are intelligently offloaded to the proxy since they will never be cached, and no data that is older than the oldest cacheable timestamp will be stored in the query's cache record.
+
+When `timeseries_eviction_method` is set to `lru`, Trickster will not calculate an oldest cacheable timestamp, but rather maintain a last-accessed time for _each timestamp_ in the cache object, and evict the Least-Recently-Used items in order to maintian the cache size.
+
+The advantage of the `oldest` methodology better cache performance, at the cost of not caching very old data. Thus, Trickster will be more performant computationally while providing a slightly lower cache hit rate. The `lru` methodology, since it requires accessing the cache on _every request_ and maintaining access times for every timestamp, is computationally more expensive, but can achieve a higher cache hit rate since it permits caching data of any age, so long as it is accessed frequently enough to avoid eviction.
+
+Most users will find the `oldest` methodology to meet their needs, so it is recommended to use `lru` only if you have a specific use case (e.g., dashboards with data from a diverse set of time ranges, where caching only relatively young data does not suffice).
diff --git a/docs/roadmap.md b/docs/roadmap.md
index b659183c9..13a6abea3 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -3,33 +3,33 @@
Our roadmap for Trickster is largely focused on a 1.0 release, which will have a completely refactored codebase. Trickster 1.0 will be more efficient and easily extensible.
Trickster 1.0 will have the following enhancements:
-- The application is refactored into Packages to simplify reuse
-- Simplified hash collision prevention and pipelining (replacing channels with mutexes)
-- Upstream Proxy interface to facilitate support for additional TSDB types
-- Support for InfluxDB acceleration
-- Full compliance with HTTP 1.0/1.1 RFC's for Proxy/Caching
-- Simpler and more efficient Delta computations
-- Caches per-origin instead of per-process
-- Size-based cache quota
-- Distributed Tracing support
+- [x] The application is refactored into Packages to simplify reuse
+- [x] Simplified hash collision prevention and pipelining (replacing channels with mutexes)
+- [x] Upstream Proxy interface to facilitate support for additional TSDB types
+- [x] Support for InfluxDB acceleration
+- [x] Simpler and more efficient Delta computations
+- [x] Caches per-origin instead of per-process
+- [x] Size-based cache quota
+- [ ] Full compliance with HTTP 1.0/1.1 RFC's for Proxy/Caching
+- [ ] Distributed Tracing support
## Timeline
### Q1 2019 - Trickster 1.0 Beta Release
-We intend to provide a Trickster 1.0 Beta Release by the end of Q1 2019 that will include the majority of features listed above.
+We intend to provide a Trickster 1.0 Beta Release by the end of Q1 2019 that will include the majority of features listed above. Our progress is indicated above via the checkboxes.
-### Q2 2019 - Trickster 1.0 GA Release
+### Q4 2019 - Trickster 1.0 GA Release
-We hope to provdie a Trickster 1.0 GA Release in the first half of Q2 2019 that includes all of the features listed above.
+We hope to provdie a Trickster 1.0 GA Release in the first half of Q4 2019 that includes all of the features listed above.
## How to Help
-You can help by contributing to Trickster 1.0 on the `next` branch, or trying it out in your environment. Docker images for the latest Trickster 1.0 Beta release will be published under the `next` tag.
+You can help by contributing to Trickster 1.0 on the `next` branch, or trying it out in your environment. Docker images for the latest Trickster 1.0 Beta release will be published under the `beta` tag.
By giving Trickster 1.0 Beta a spin, you can help us identify and fix defects more quickly. Be sure to file issues if you find something wrong, using the `1.0` label. If you can reliably reproduce the issue, provide detailed steps so that developers can more easily root-cause the issue.
-If you want to contribute to Trickster 1.0, take any of the issues labeled `1.0` that are not already assigned. Many of these have been outstanding for some time pending the Interface model, so now is great time to look at extending Trickster to work with your TSDB of choice.
+If you want to contribute to Trickster 1.0, take any of the issues labeled `1.0 Release` or `1.x Release` that are not already assigned. Many of these have been outstanding for some time, pending the Interface model, so now is great time to look at extending Trickster to work with your TSDB of choice.
## Thank You
diff --git a/docs/supported-origin-types.md b/docs/supported-origin-types.md
new file mode 100644
index 000000000..ac6dc2228
--- /dev/null
+++ b/docs/supported-origin-types.md
@@ -0,0 +1,33 @@
+# Supported Origin Types
+
+Trickster currently supports the following Origin Types:
+
+### 
Generic HTTP Reverse Proxy Cache _(Currently Experimental)_
+
+Trickster operates as a fully-featured and highly-customizable reverse proxy cache, designed to accellerate and scale upstream endpoints like API services and other simple http services. Specify `'reverseproxycache'` or just `'rpc'` as the Origin Type when configuring Trickster.
+
+---
+
+## Time Series Databases
+
+### 
Prometheus
+
+Trickster fully supports the [Prometheus HTTP API (v1)](https://prometheus.io/docs/prometheus/latest/querying/api/). Specify `'prometheus'` as the Origin Type when configuring Trickster.
+
+### 
InfluxDB _(Currently Experimental)_
+
+Trickster 1.0 has experimental support for InfluxDB. Specify `'influxdb'` as the Origin Type when configuring Trickster.
+
+See the [InfluxDB Support Document](./influxdb.md) for more information.
+
+### 
ClickHouse _(Currently Experimental)_
+
+Trickster 1.0 has experimental support for ClickHouse. Specify `'clickhouse'` as the Origin Type when configuring Trickster.
+
+See the [ClickHouse Support Document](./clickhouse.md) for more information.
+
+### 
Circonus IRONdb _(Currently Experimental)_
+
+Experimental support has been included for the Circonus IRONdb time-series database. If Grafana is used for visualizations, the Circonus IRONdb data source plug-in for Grafana can be configured to use Trickster as its data source. All IRONdb data retrieval operations, including CAQL queries, are supported.
+
+When configuring an IRONdb origin, specify `'irondb'` as the origin type in the Trickster configuration. The `host` value can be set directly to the address and port of an IRONdb node, but it is recommended to use the Circonus API proxy service. When using the proxy service, set the `host` value to the address and port of the proxy service, and set the `api_path` value to `'irondb'`.
diff --git a/docs/tls.md b/docs/tls.md
new file mode 100644
index 000000000..ebf6b62de
--- /dev/null
+++ b/docs/tls.md
@@ -0,0 +1,52 @@
+# TLS Support
+
+Trickster supports TLS on both the frontend server and backend clients.
+
+## Basics
+
+To enable the TLS server, you must specify the `tls_listen_port`, and optionally, the `tls_listen_address` in the `[proxy_server]` section of your config file. For example:
+
+```toml
+[proxy_server]
+
+listen_port = 9090
+tls_listen_port = 8443
+```
+
+Note, Trickster will only start listening on the TLS port if at least one origin has a valid certificate and key configured.
+
+Each origin section of a Trickster config file can be augmented with the optional `tls` section to modify TLS behavior for front-end and back-end requests. For example:
+
+```toml
+[origins]
+
+ [origins.example] # origin for example
+
+ [origins.example.tls] # TLS settigs for origin named example
+ # front-end configs
+ full_chain_cert_path = '/path/to/my/cert.pem'
+ private_key_path = '/path/to/my/key.pem'
+ # back-end configs
+ insecure_skip_verify = true
+ certificate_authority_paths = [ '/path/to/ca1.pem', '/path/to/ca2.pem' ]
+ client_cert_path = '/path/to/client/cert.pem'
+ client_key_path = '/path/to/client/key.pem'
+```
+
+## Front-End
+
+Each origin can handle encryption with exactly 1 certificate and key pair, as configured in the TLS section of the origin config (demonstrated above).
+
+If the path to any configured Certificate or Key file is unreachable or unparsable, Trickster will exit upon startup with an error providing reasonable context.
+
+You may use the same TLS certificate and key for multiple origins, depending upon how your Trickster configurations are laid out. Any certificates configured by Trickster must match the hostname header of the inbound http request (exactly, or by wildcard interpolation), or clients will likely reject the certificate for security issues.
+
+## Back-End
+
+Each Trickster origin front-end configuration is paired with its own back-end http(s) client, which can be configured in the TLS section of the origin config, as demonstrated above.
+
+`insecure_skip_verify` will instruct the http client to ignore hostname verification issues with the upstream origin's certificate, and process the request anyway. This is analogous to `-k | --insecure` in curl.
+
+`certificate_authority_paths` will provide the http client with a list of certificate authorities (used in addition to any OS-provided root CA's) to use when determining the trust of an upstream origin's tls certificate. In all cases, the Root CA's installed to the operating system on which Trickster is running are used for trust by the client.
+
+To us Mutual Authentication with an upstream origin server, configure Trickster with Client Certificates using `client_cert_path` and `client_key_path` parameters, as shown above. You will likely need to also configure a custom CA in `certificate_authority_paths` to represent your certificate signer, unless it has been added to the underlying Operating System's CA list.
diff --git a/filesystem.go b/filesystem.go
deleted file mode 100644
index 3f9f41a7b..000000000
--- a/filesystem.go
+++ /dev/null
@@ -1,168 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "fmt"
- "io/ioutil"
- "os"
- "strconv"
- "strings"
- "sync"
- "time"
-
- "github.com/go-kit/kit/log/level"
- "golang.org/x/sys/unix"
-)
-
-// FilesystemCache describes a Filesystem Cache
-type FilesystemCache struct {
- T *TricksterHandler
- Config FilesystemCacheConfig
- mutexes map[string]*sync.Mutex
- mapMutex sync.Mutex
-}
-
-// Connect instantiates the FilesystemCache mutex map and starts the Expired Entry Reaper goroutine
-func (c *FilesystemCache) Connect() error {
- level.Info(c.T.Logger).Log("event", "filesystem cache setup", "cachePath", c.Config.CachePath)
-
- if err := makeDirectory(c.Config.CachePath); err != nil {
- return err
- }
-
- c.mutexes = make(map[string]*sync.Mutex)
-
- go c.Reap()
- return nil
-}
-
-// Store places an object in the cache using the specified key and ttl
-func (c *FilesystemCache) Store(cacheKey string, data string, ttl int64) error {
- expFile, dataFile := c.getFileNames(cacheKey)
- expiration := []byte(strconv.FormatInt(time.Now().Unix()+ttl, 10))
-
- level.Debug(c.T.Logger).Log("event", "filesystem cache store", "key", cacheKey, "expFile", expFile, "dataFile", dataFile)
- mtx := c.getMutex(cacheKey)
- mtx.Lock()
- err1 := ioutil.WriteFile(dataFile, []byte(data), os.FileMode(0777))
- err2 := ioutil.WriteFile(expFile, expiration, os.FileMode(0777))
- mtx.Unlock()
-
- if err1 != nil {
- return err1
- } else if err2 != nil {
- return err2
- }
- return nil
-}
-
-// Retrieve looks for an object in cache and returns it (or an error if not found)
-func (c *FilesystemCache) Retrieve(cacheKey string) (string, error) {
- _, dataFile := c.getFileNames(cacheKey)
- level.Debug(c.T.Logger).Log("event", "filesystem cache retrieve", "key", cacheKey, "dataFile", dataFile)
-
- mtx := c.getMutex(cacheKey)
- mtx.Lock()
- content, err := ioutil.ReadFile(dataFile)
- mtx.Unlock()
- if err != nil {
- return "", fmt.Errorf("Value for key [%s] not in cache", cacheKey)
- }
-
- return string(content), nil
-}
-
-// Reap continually iterates through the cache to find expired elements and removes them
-func (c *FilesystemCache) Reap() {
- for {
- now := time.Now().Unix()
-
- files, err := ioutil.ReadDir(c.Config.CachePath)
- if err == nil {
- for _, file := range files {
- if strings.HasSuffix(file.Name(), ".expiration") {
- cacheKey := strings.Replace(file.Name(), ".expiration", "", 1)
- expFile, dataFile := c.getFileNames(cacheKey)
- mtx := c.getMutex(cacheKey)
- mtx.Lock()
- content, err := ioutil.ReadFile(expFile)
- if err == nil {
- expiration, err := strconv.ParseInt(string(content), 10, 64)
- if err != nil || expiration < now {
- level.Debug(c.T.Logger).Log("event", "filesystem cache reap", "key", cacheKey, "dataFile", dataFile)
-
- // Get a lock
- c.T.ChannelCreateMtx.Lock()
-
- // Delete the key
- os.Remove(expFile)
- os.Remove(dataFile)
-
- // Close out the channel if it exists
- if _, ok := c.T.ResponseChannels[cacheKey]; ok {
- close(c.T.ResponseChannels[cacheKey])
- delete(c.T.ResponseChannels, cacheKey)
- }
-
- // Unlock
- c.T.ChannelCreateMtx.Unlock()
- }
- }
- mtx.Unlock()
- }
- }
- }
-
- time.Sleep(time.Duration(c.T.Config.Caching.ReapSleepMS) * time.Millisecond)
- }
-}
-
-// Close is not used for FilesystemCache
-func (c *FilesystemCache) Close() error {
- return nil
-}
-
-func (c *FilesystemCache) getFileNames(cacheKey string) (string, string) {
- prefix := strings.Replace(c.Config.CachePath+"/"+cacheKey+".", "//", "/", 1)
- return prefix + "expiration", prefix + "data"
-}
-
-func (c *FilesystemCache) getMutex(cacheKey string) *sync.Mutex {
- var mtx *sync.Mutex
- var ok bool
- c.mapMutex.Lock()
- if mtx, ok = c.mutexes[cacheKey]; !ok {
- mtx = &sync.Mutex{}
- c.mutexes[cacheKey] = mtx
- }
- c.mapMutex.Unlock()
-
- return mtx
-}
-
-// writeable returns true if the path is writeable by the calling process.
-func writeable(path string) bool {
- return unix.Access(path, unix.W_OK) == nil
-}
-
-// makeDirectory creates a directory on the filesystem and exits the application in the event of a failure.
-func makeDirectory(path string) error {
- err := os.MkdirAll(path, 0755)
- if err != nil || !writeable(path) {
- return fmt.Errorf("[%s] directory is not writeable by the trickster: %v", path, err)
- }
-
- return nil
-}
diff --git a/filesystem_test.go b/filesystem_test.go
deleted file mode 100644
index 7a05cb4a4..000000000
--- a/filesystem_test.go
+++ /dev/null
@@ -1,73 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "testing"
-
- "github.com/go-kit/kit/log"
-)
-
-func TestFilesystemCache_Connect(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- fc := FilesystemCache{T: &tr, Config: FilesystemCacheConfig{CachePath: "."}}
-
- // it should connect
- err := fc.Connect()
- if err != nil {
- t.Error(err)
- }
-}
-
-func TestFilesystemCache_Store(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- fc := FilesystemCache{T: &tr, Config: FilesystemCacheConfig{CachePath: "."}}
-
- err := fc.Connect()
- if err != nil {
- t.Error(err)
- }
-
- // it should store a value
- err = fc.Store("cacheKey", "data", 60000)
- if err != nil {
- t.Error(err)
- }
-}
-
-func TestFilesystemCache_Retrieve(t *testing.T) {
- cfg := Config{Caching: CachingConfig{ReapSleepMS: 1}}
- tr := TricksterHandler{Logger: log.NewNopLogger(), Config: &cfg}
- fc := FilesystemCache{T: &tr, Config: FilesystemCacheConfig{CachePath: "."}}
-
- err := fc.Connect()
- if err != nil {
- t.Error(err)
- }
- err = fc.Store("cacheKey", "data", 60000)
- if err != nil {
- t.Error(err)
- }
-
- // it should retrieve a value
- data, err := fc.Retrieve("cacheKey")
- if err != nil {
- t.Error(err)
- }
- if data != "data" {
- t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
- }
-}
diff --git a/flags.go b/flags.go
deleted file mode 100644
index c313a81f5..000000000
--- a/flags.go
+++ /dev/null
@@ -1,159 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "flag"
- "fmt"
- "io/ioutil"
- "os"
- "strconv"
-)
-
-const (
- // Command-line flags
- cfConfig = "config"
- cfVersion = "version"
- cfLogLevel = "log-level"
- cfInstanceID = "instance-id"
- cfOrigin = "origin"
- cfProxyPort = "proxy-port"
- cfMetricsPort = "metrics-port"
- cfProfilerPort = "profiler-port"
-
- // Environment variables
- evOrigin = "TRK_ORIGIN"
- evProxyPort = "TRK_PROXY_PORT"
- evMetricsPort = "TRK_METRICS_PORT"
- evLogLevel = "TRK_LOG_LEVEL"
- evProfilerPort = "TRK_PROFILER_PORT"
-)
-
-// loadConfiguration reads the config path from Flags,
-// Loads the configs (w/ default values where missing)
-// and then evaluates any provided flags as overrides
-func loadConfiguration(c *Config, arguments []string) error {
- var path string
- var version bool
-
- f := flag.NewFlagSet(applicationName, -1)
- f.SetOutput(ioutil.Discard)
- f.StringVar(&path, cfConfig, "", "Supplies Path to Config File")
- f.BoolVar(&version, cfVersion, false, "Prints trickster version")
- f.Parse(arguments)
-
- // If the config file is not specified on the cmdline then try the default
- // location to load the config file. If the default config does not exist
- // then move on, no big deal.
- if path != "" {
- if err := c.LoadFile(path); err != nil {
- return err
- }
- } else {
- _, err := os.Open(c.Main.ConfigFile)
- if err == nil {
- if err := c.LoadFile(c.Main.ConfigFile); err != nil {
- return err
- }
- }
- }
-
- // Display version information then exit the program
- if version == true {
- fmt.Println(applicationVersion)
- os.Exit(3)
- }
-
- // Load from Environment Variables
- loadEnvVars(c)
-
- //Load from command line flags.
- loadFlags(c, arguments)
-
- return nil
-}
-
-func loadEnvVars(c *Config) {
- // Origin
- if x := os.Getenv(evOrigin); x != "" {
- c.DefaultOriginURL = x
- }
-
- // Proxy Port
- if x := os.Getenv(evProxyPort); x != "" {
- if y, err := strconv.ParseInt(x, 10, 64); err == nil {
- c.ProxyServer.ListenPort = int(y)
- }
- }
-
- // Metrics Port
- if x := os.Getenv(evMetricsPort); x != "" {
- if y, err := strconv.ParseInt(x, 10, 64); err == nil {
- c.Metrics.ListenPort = int(y)
- }
- }
-
- // ProfilerPort
- if x := os.Getenv(evProfilerPort); x != "" {
- if y, err := strconv.ParseInt(x, 10, 64); err == nil {
- c.Profiler.ListenPort = int(y)
- c.Profiler.Enabled = true
- }
- }
-
- // LogLevel
- if x := os.Getenv(evLogLevel); x != "" {
- c.Logging.LogLevel = x
- }
-
-}
-
-// loadFlags loads configuration from command line flags.
-func loadFlags(c *Config, arguments []string) {
- var path string
- var version bool
- var origin string
- var proxyListenPort int
- var metricsListenPort int
- var profilerListenPort int
-
- f := flag.NewFlagSet(applicationName, flag.ExitOnError)
- f.BoolVar(&version, cfVersion, true, "Prints Trickster version")
- f.StringVar(&c.Logging.LogLevel, cfLogLevel, c.Logging.LogLevel, "Level of Logging to use (debug, info, warn, error)")
- f.IntVar(&c.Main.InstanceID, cfInstanceID, 0, "Instance ID for when running multiple processes")
- f.StringVar(&origin, cfOrigin, "", "URL to the Prometheus Origin. Enter it like you would in grafana, e.g., http://prometheus:9090")
- f.IntVar(&proxyListenPort, cfProxyPort, 0, "Port that the Proxy server will listen on.")
- f.IntVar(&metricsListenPort, cfMetricsPort, 0, "Port that the /metrics endpoint will listen on.")
- f.IntVar(&profilerListenPort, cfProfilerPort, 0, "Port that the /debug/pprof endpoint will listen on.")
-
- // BEGIN IGNORED FLAGS
- f.StringVar(&path, cfConfig, "", "Path to Trickster Config File")
- // END IGNORED FLAGS
-
- f.Parse(arguments)
-
- if len(origin) > 0 {
- c.DefaultOriginURL = origin
- }
- if proxyListenPort > 0 {
- c.ProxyServer.ListenPort = proxyListenPort
- }
- if metricsListenPort > 0 {
- c.Metrics.ListenPort = metricsListenPort
- }
- if profilerListenPort > 0 {
- c.Profiler.ListenPort = profilerListenPort
- c.Profiler.Enabled = true
- }
-}
diff --git a/go.mod b/go.mod
index c6d09d8b4..5591772c0 100644
--- a/go.mod
+++ b/go.mod
@@ -1,28 +1,44 @@
module github.com/Comcast/trickster
+replace gotest.tools => github.com/gotestyourself/gotest.tools v2.2.0+incompatible
+
require (
+ github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect
github.com/BurntSushi/toml v0.3.1
+ github.com/alecthomas/gometalinter v3.0.0+incompatible // indirect
github.com/alicebob/gopher-json v0.0.0-20180125190556-5a6b3ba71ee6 // indirect
- github.com/alicebob/miniredis v0.0.0-20181205055656-cfad8aca71cc
- github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 // indirect
- github.com/coreos/bbolt v1.3.0
- github.com/go-kit/kit v0.8.0
- github.com/go-logfmt/logfmt v0.4.0 // indirect
- github.com/go-redis/redis v6.14.2+incompatible
+ github.com/alicebob/miniredis v2.5.0+incompatible
+ github.com/boltdb/bolt v1.3.1 // indirect
+ github.com/coreos/bbolt v1.3.3
+ github.com/dgraph-io/badger v1.6.0
+ github.com/dgryski/go-farm v0.0.0-20191112170834-c2139c5d712b // indirect
+ github.com/fatih/color v1.7.0 // indirect
+ github.com/go-kit/kit v0.9.0
+ github.com/go-logfmt/logfmt v0.5.0 // indirect
+ github.com/go-redis/redis v6.15.6+incompatible
github.com/go-stack/stack v1.8.0
- github.com/golang/protobuf v1.2.0 // indirect
- github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
+ github.com/gojp/goreportcard v0.0.0-20200102082842-9f8184163627 // indirect
+ github.com/golang/snappy v0.0.1
github.com/gomodule/redigo v2.0.0+incompatible // indirect
- github.com/gorilla/context v1.1.1 // indirect
- github.com/gorilla/handlers v1.4.0
- github.com/gorilla/mux v1.6.2
- github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
- github.com/pkg/errors v0.8.0
- github.com/prometheus/client_golang v0.9.1
- github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 // indirect
- github.com/prometheus/common v0.0.0-20181126121408-4724e9255275
- github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a // indirect
- github.com/yuin/gopher-lua v0.0.0-20181109042959-a0dfe84f6227 // indirect
- golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a
- gopkg.in/natefinch/lumberjack.v2 v2.0.0-20170531160350-a96e63847dc3
+ github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
+ github.com/gorilla/handlers v1.4.2
+ github.com/gorilla/mux v1.7.3
+ github.com/influxdata/influxdb v1.7.9
+ github.com/mattn/go-colorable v0.1.4 // indirect
+ github.com/mattn/go-isatty v0.0.10 // indirect
+ github.com/mattn/goveralls v0.0.4 // indirect
+ github.com/philhofer/fwd v1.0.0 // indirect
+ github.com/pkg/errors v0.9.1 // indirect
+ github.com/prometheus/client_golang v1.3.0
+ github.com/prometheus/common v0.8.0
+ github.com/rakyll/gotest v0.0.0-20191108192113-45d501058f2a // indirect
+ github.com/tinylib/msgp v1.1.1
+ github.com/yuin/gopher-lua v0.0.0-20190514113301-1cd887cd7036 // indirect
+ golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa
+ golang.org/x/sys v0.0.0-20200116001909-b77594299b42
+ golang.org/x/tools v0.0.0-20191206204035-259af5ff87bd // indirect
+ gopkg.in/alecthomas/kingpin.v3-unstable v3.0.0-20191105091915-95d230a53780 // indirect
+ gopkg.in/natefinch/lumberjack.v2 v2.0.0
)
+
+go 1.13
diff --git a/go.sum b/go.sum
index a8ab84058..bdc881cf8 100644
--- a/go.sum
+++ b/go.sum
@@ -1,51 +1,226 @@
+github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9 h1:HD8gA2tkByhMAwYaFAX9w2l7vxvBQ5NMoxDrkhqhtn4=
+github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8=
+github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 h1:cTp8I5+VIoKjsnZuH8vjyaysT/ses3EvZeaV/1UkF2M=
+github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/alecthomas/gometalinter v3.0.0+incompatible h1:e9Zfvfytsw/e6Kd/PYd75wggK+/kX5Xn8IYDUKyc5fU=
+github.com/alecthomas/gometalinter v3.0.0+incompatible/go.mod h1:qfIpQGGz3d+NmgyPBqv+LSh50emm1pt72EtcX2vKYQk=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 h1:Hs82Z41s6SdL1CELW+XaDYmOH4hkBN4/N9og/AsOv7E=
+github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alicebob/gopher-json v0.0.0-20180125190556-5a6b3ba71ee6 h1:45bxf7AZMwWcqkLzDAQugVEwedisr5nRJ1r+7LYnv0U=
github.com/alicebob/gopher-json v0.0.0-20180125190556-5a6b3ba71ee6/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
-github.com/alicebob/miniredis v0.0.0-20181205055656-cfad8aca71cc h1:oajiik5ClkUDgM2JOF9hI7muk5/jI+FOCa7YDpFqmGY=
-github.com/alicebob/miniredis v0.0.0-20181205055656-cfad8aca71cc/go.mod h1:8HZjEj4yU0dwhYHky+DxYx+6BMjkBbe5ONFIF1MXffk=
-github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0=
+github.com/alicebob/miniredis v2.5.0+incompatible h1:yBHoLpsyjupjz3NL3MhKMVkR41j82Yjf3KFv7ApYzUI=
+github.com/alicebob/miniredis v2.5.0+incompatible/go.mod h1:8HZjEj4yU0dwhYHky+DxYx+6BMjkBbe5ONFIF1MXffk=
+github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
+github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
+github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
+github.com/cespare/xxhash/v2 v2.1.0 h1:yTUvW7Vhb89inJ+8irsUqiWjh8iT6sQPZiQzI6ReGkA=
+github.com/cespare/xxhash/v2 v2.1.0/go.mod h1:dgIUBU3pDso/gPgZ1osOZ0iQf77oPR28Tjxl5dIMyVM=
+github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
+github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
-github.com/coreos/bbolt v1.3.0 h1:HIgH5xUWXT914HCI671AxuTTqjj64UOFr7pHn48LUTI=
-github.com/coreos/bbolt v1.3.0/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
-github.com/go-kit/kit v0.8.0 h1:Wz+5lgoB0kkuqLEc6NVmwRknTKP6dTGbSqvhZtBI/j0=
+github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
+github.com/coreos/bbolt v1.3.3 h1:n6AiVyVRKQFNb6mJlwESEvvLoDyiTzXX7ORAUlkeBdY=
+github.com/coreos/bbolt v1.3.3/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
+github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
+github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dgraph-io/badger v1.6.0 h1:DshxFxZWXUcO0xX476VJC07Xsr6ZCBVRHKZ93Oh7Evo=
+github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4=
+github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 h1:tdlZCpZ/P9DhczCTSixgIKmwPv6+wP5DGjqLYw5SUiA=
+github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
+github.com/dgryski/go-farm v0.0.0-20191112170834-c2139c5d712b h1:SeiGBzKrEtuDddnBABHkp4kq9sBGE9nuYmk6FPTg0zg=
+github.com/dgryski/go-farm v0.0.0-20191112170834-c2139c5d712b/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
+github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
+github.com/go-kit/kit v0.9.0 h1:wDJmvq38kDhkVxi50ni9ykkdUr1PKgqKOoi01fa0Mdk=
+github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
+github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0 h1:MP4Eh7ZCb31lleYCFuwm0oe4/YGak+5l1vA2NOE80nA=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
-github.com/go-redis/redis v6.14.2+incompatible h1:UE9pLhzmWf+xHNmZsoccjXosPicuiNaInPgym8nzfg0=
-github.com/go-redis/redis v6.14.2+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
+github.com/go-logfmt/logfmt v0.5.0 h1:TrB8swr/68K7m9CcGut2g3UOihhbcbiMAYiuTXdEih4=
+github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
+github.com/go-redis/redis v6.15.5+incompatible h1:pLky8I0rgiblWfa8C1EV7fPEUv0aH6vKRaYHc/YRHVk=
+github.com/go-redis/redis v6.15.5+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
+github.com/go-redis/redis v6.15.6+incompatible h1:H9evprGPLI8+ci7fxQx6WNZHJSb7be8FqJQRhdQZ5Sg=
+github.com/go-redis/redis v6.15.6+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
-github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM=
+github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/gojp/goreportcard v0.0.0-20200102082842-9f8184163627 h1:g5bGbrCY46a0KjewTqkYB8PtV9r6TQPrX34CIy+6OdQ=
+github.com/gojp/goreportcard v0.0.0-20200102082842-9f8184163627/go.mod h1:/DA2Xpp+OaR3EHafQSnT9SKOfbG2NPQR/qp6Qr8AgIw=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db h1:woRePGFeVFfLKN/pOkfl+p/TAqKOfFu+7KPlMVpok/w=
-github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
+github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
+github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/gomodule/redigo v2.0.0+incompatible h1:K/R+8tc58AaqLkqG2Ol3Qk+DR/TlNuhuh457pBFPtt0=
github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
-github.com/gorilla/context v1.1.1 h1:AWwleXJkX/nhcU9bZSnZoi3h/qGYqQAGhq6zZe/aQW8=
-github.com/gorilla/context v1.1.1/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg=
-github.com/gorilla/handlers v1.4.0 h1:XulKRWSQK5uChr4pEgSE4Tc/OcmnU9GJuSwdog/tZsA=
-github.com/gorilla/handlers v1.4.0/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ=
-github.com/gorilla/mux v1.6.2 h1:Pgr17XVTNXAk3q/r4CpKzC5xBM/qW1uVLV+IhRZpIIk=
-github.com/gorilla/mux v1.6.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
+github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
+github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
+github.com/gorilla/handlers v1.4.2 h1:0QniY0USkHQ1RGCLfKxeNHK9bkDHGRYGNDFBCS+YARg=
+github.com/gorilla/handlers v1.4.2/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ=
+github.com/gorilla/mux v1.7.3 h1:gnP5JzjVOuiZD07fKKToCAOjS0yOpj/qPETTXCCS6hw=
+github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
+github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
+github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/influxdata/influxdb v1.7.8 h1:oXd5TjXzU1b+xyFaH/8Ij+nCoUgyuO3ZDpgCuo62yg0=
+github.com/influxdata/influxdb v1.7.8/go.mod h1:qZna6X/4elxqT3yI9iZYdZrWWdeFOOprn86kgg4+IzY=
+github.com/influxdata/influxdb v1.7.9 h1:uSeBTNO4rBkbp1Be5FKRsAmglM9nlx25TzVQRQt1An4=
+github.com/influxdata/influxdb v1.7.9/go.mod h1:qZna6X/4elxqT3yI9iZYdZrWWdeFOOprn86kgg4+IzY=
+github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
+github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515 h1:T+h1c/A9Gawja4Y9mFVWj2vyii2bbUNDw3kt9VxK2EY=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
+github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
+github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
+github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
+github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
+github.com/mattn/go-isatty v0.0.10 h1:qxFzApOv4WsAL965uUPIsXzAKCZxN2p9UqdhFS4ZW10=
+github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
+github.com/mattn/goveralls v0.0.4 h1:/mdWfiU2y8kZ48EtgByYev/XT3W4dkTuKLOJJsh/r+o=
+github.com/mattn/goveralls v0.0.4/go.mod h1:8d1ZMHsd7fW6IRPKQh46F2WRpyib5/X4FOpevwGNQEw=
github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
-github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/nicksnyder/go-i18n v2.0.3+incompatible h1:XCCaWsCoy4KlWkhOr+63dkv6oJmitJ573uJqDBAiFiQ=
+github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/philhofer/fwd v1.0.0 h1:UbZqGr5Y38ApvM/V/jEljVxwocdweyH+vmYvRPBnbqQ=
+github.com/philhofer/fwd v1.0.0/go.mod h1:gk3iGcWd9+svBvR0sR+KPcfE+RNWozjowpeBVG3ZVNU=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
-github.com/prometheus/client_golang v0.9.1 h1:K47Rk0v/fkEfwfQet2KWhscE0cJzjgCCDBG2KHZoVno=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
-github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 h1:idejC8f05m9MGOsuEi1ATq9shN03HrxNkD/luQvxCv8=
+github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
+github.com/prometheus/client_golang v1.1.0 h1:BQ53HtBmfOitExawJ6LokA4x8ov/z0SYYb0+HxJfRI8=
+github.com/prometheus/client_golang v1.1.0/go.mod h1:I1FGZT9+L76gKKOs5djB6ezCbFQP1xR9D75/vuwEF3g=
+github.com/prometheus/client_golang v1.2.1 h1:JnMpQc6ppsNgw9QPAGF6Dod479itz7lvlsMzzNayLOI=
+github.com/prometheus/client_golang v1.2.1/go.mod h1:XMU6Z2MjaRKVu/dC1qupJI9SiNkDYzz3xecMgSW/F+U=
+github.com/prometheus/client_golang v1.3.0 h1:miYCvYqFXtl/J9FIy8eNpBfYthAEFg+Ys0XyUVEcDsc=
+github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeDPbaTKGT+JTgUa3og=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
-github.com/prometheus/common v0.0.0-20181126121408-4724e9255275 h1:PnBWHBf+6L0jOqq0gIVUe6Yk0/QMZ640k6NvkxcBf+8=
-github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
-github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a h1:9a8MnZMP0X2nLJdBg+pBmGgkJlSaKC2KaQmTCk1XDtE=
-github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
-github.com/yuin/gopher-lua v0.0.0-20181109042959-a0dfe84f6227 h1:GRy+0tGtORsCA+CJUMfhLuN71eQ0LtsQRDBQKbzESdc=
-github.com/yuin/gopher-lua v0.0.0-20181109042959-a0dfe84f6227/go.mod h1:fFiAh+CowNFr0NK5VASokuwKwkbacRmHsVA7Yb1Tqac=
-golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a h1:1n5lsVfiQW3yfsRGu98756EH1YthsFqr/5mxHduZW2A=
+github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 h1:S/YWwWx/RA8rT8tKFRuGUZhuA90OyIBpPCXkcbwU8DE=
+github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 h1:gQz4mCbXsO+nc9n1hCxHcGA3Zx3Eo+UHZoInFGUIXNM=
+github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/client_model v0.1.0 h1:ElTg5tNp4DqfV7UQjDqv2+RJlNzsDtvNAWccbItceIE=
+github.com/prometheus/client_model v0.1.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
+github.com/prometheus/common v0.6.0 h1:kRhiuYSXR3+uv2IbVbZhUxK5zVD/2pp3Gd2PpvPkpEo=
+github.com/prometheus/common v0.6.0/go.mod h1:eBmuwkDJBwy6iBfxCBob6t6dR6ENT/y+J+Zk0j9GMYc=
+github.com/prometheus/common v0.7.0 h1:L+1lyG48J1zAQXA3RBX/nG/B3gjlHq0zTt2tlbJLyCY=
+github.com/prometheus/common v0.7.0/go.mod h1:DjGbpBbp5NYNiECxcL/VnbXCCaQpKd3tt26CguLLsqA=
+github.com/prometheus/common v0.8.0 h1:bLkjvFe2ZRX1DpcgZcdf7j/+MnusEps5hktST/FHA34=
+github.com/prometheus/common v0.8.0/go.mod h1:PC/OgXc+UN7B4ALwvn1yzVZmVwvhXp5JsbBv6wSv6i0=
+github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
+github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
+github.com/prometheus/procfs v0.0.3 h1:CTwfnzjQ+8dS6MhHHu4YswVAD99sL2wjPqP+VkURmKE=
+github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
+github.com/prometheus/procfs v0.0.5 h1:3+auTFlqw+ZaQYJARz6ArODtkaIwtvBTx3N2NehQlL8=
+github.com/prometheus/procfs v0.0.5/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
+github.com/prometheus/procfs v0.0.8 h1:+fpWZdT24pJBiqJdAwYBjPSk+5YmQzYNPYzQsdzLkt8=
+github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
+github.com/rakyll/gotest v0.0.0-20191108192113-45d501058f2a h1:/kX+lZpr87Pb0yJKyxW40ZZO6jl52jFMmoQ0YhfwGLM=
+github.com/rakyll/gotest v0.0.0-20191108192113-45d501058f2a/go.mod h1:jpFrc1UTqK0FtfF3doi3pEUBgWHYELkOPPECUlDsM2Q=
+github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
+github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
+github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
+github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
+github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/tinylib/msgp v1.1.0 h1:9fQd+ICuRIu/ue4vxJZu6/LzxN0HwMds2nq/0cFvxHU=
+github.com/tinylib/msgp v1.1.0/go.mod h1:+d+yLhGm8mzTaHzB+wgMYrodPfmZrzkirds8fDWklFE=
+github.com/tinylib/msgp v1.1.1 h1:TnCZ3FIuKeaIy+F45+Cnp+caqdXGy4z74HvwXN+570Y=
+github.com/tinylib/msgp v1.1.1/go.mod h1:+d+yLhGm8mzTaHzB+wgMYrodPfmZrzkirds8fDWklFE=
+github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
+github.com/yuin/gopher-lua v0.0.0-20190514113301-1cd887cd7036 h1:1b6PAtenNyhsmo/NKXVe34h7JEZKva1YB/ne7K7mqKM=
+github.com/yuin/gopher-lua v0.0.0-20190514113301-1cd887cd7036/go.mod h1:gqRgreBUhTSL0GeU64rtZ3Uq3wtjOa/TB2YfrtkCbVQ=
+golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297 h1:k7pJ2yAPLPgbskkFdhRCsA77k2fySZ1zf2zCjvQCiIM=
+golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20191021124707-24d2ffbea1e8 h1:L4W1teiyF4Jl6VuapLNV/LYho36udiBQsfbNu7eRMeo=
+golang.org/x/net v0.0.0-20191021124707-24d2ffbea1e8/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa h1:F+8P+gmewFQYRk6JoLQLwjBCTu3mcIURZfNkVweuRKA=
+golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-gopkg.in/natefinch/lumberjack.v2 v2.0.0-20170531160350-a96e63847dc3 h1:AFxeG48hTWHhDTQDk/m2gorfVHUEa9vo3tp3D7TzwjI=
-gopkg.in/natefinch/lumberjack.v2 v2.0.0-20170531160350-a96e63847dc3/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
+golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190801041406-cbf593c0f2f3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190904154756-749cb33beabd h1:DBH9mDw0zluJT/R+nGuV3jWFWLFaHyYZWD4tOT+cjn0=
+golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191020212454-3e7259c5e7c2 h1:nq114VpM8lsSlP+lyUbANecYHYiFcSNFtqcBlxRV+gA=
+golang.org/x/sys v0.0.0-20191020212454-3e7259c5e7c2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191206220618-eeba5f6aabab h1:FvshnhkKW+LO3HWHodML8kuVX8rnJTxKm9dFPuI68UM=
+golang.org/x/sys v0.0.0-20191206220618-eeba5f6aabab/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200116001909-b77594299b42 h1:vEOn+mP2zCOVzKckCZy6YsCtDblrpj/w7B9nxGNELpg=
+golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/tools v0.0.0-20191206204035-259af5ff87bd h1:Zc7EU2PqpsNeIfOoVA7hvQX4cS3YDJEs5KlfatT3hLo=
+golang.org/x/tools v0.0.0-20191206204035-259af5ff87bd/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/alecthomas/kingpin.v3-unstable v3.0.0-20191105091915-95d230a53780 h1:CEBpW6C191eozfEuWdUmIAHn7lwlLxJ7HVdr2e2Tsrw=
+gopkg.in/alecthomas/kingpin.v3-unstable v3.0.0-20191105091915-95d230a53780/go.mod h1:3HH7i1SgMqlzxCcBmUHW657sD4Kvv9sC3HpL3YukzwA=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
diff --git a/handlers.go b/handlers.go
deleted file mode 100644
index fb53cbc6c..000000000
--- a/handlers.go
+++ /dev/null
@@ -1,1208 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "crypto/md5"
- "encoding/json"
- "fmt"
- "io/ioutil"
- "math"
- "net/http"
- "net/url"
- "regexp"
- "strconv"
- "strings"
- "sync"
- "time"
-
- "github.com/go-kit/kit/log"
- "github.com/go-kit/kit/log/level"
- "github.com/golang/snappy"
- "github.com/gorilla/mux"
- "github.com/pkg/errors"
- "github.com/prometheus/common/model"
-)
-
-const (
- // Origin database types
- otPrometheus = "prometheus"
-
- // Common HTTP Header Values
- hvNoCache = "no-cache"
- hvApplicationJSON = "application/json"
-
- // Common HTTP Header Names
- hnCacheControl = "Cache-Control"
- hnAllowOrigin = "Access-Control-Allow-Origin"
- hnContentType = "Content-Type"
- hnAuthorization = "Authorization"
-
- // HTTP methods
- hmGet = "GET"
-
- // Prometheus response values
- rvSuccess = "success"
- rvMatrix = "matrix"
- rvVector = "vector"
-
- // Common URL parameter names
- upQuery = "query"
- upStart = "start"
- upEnd = "end"
- upStep = "step"
- upOriginFqdn = "origin_fqdn"
- upOriginPort = "origin_port"
- upTimeout = "timeout"
- upOrigin = "origin"
- upTime = "time"
-
- // Cache lookup results
- crKeyMiss = "kmiss"
- crRangeMiss = "rmiss"
- crHit = "hit"
- crPartialHit = "phit"
- crPurge = "purge"
-)
-
-// TricksterHandler contains the services the Handlers need to operate
-type TricksterHandler struct {
- Logger log.Logger
- Config *Config
- Metrics *ApplicationMetrics
- Cacher Cache
- ResponseChannels map[string]chan *ClientRequestContext
- ChannelCreateMtx sync.Mutex
-}
-
-// HTTP Handlers
-
-// pingHandler handles calls to /ping, which checks the health of the Trickster app, but not connectivity to upstream origins
-// it respond with 200 OK and "pong" so long as the HTTP Server is running and taking requests
-func (t *TricksterHandler) pingHandler(w http.ResponseWriter, r *http.Request) {
- w.Header().Set(hnCacheControl, hvNoCache)
- w.WriteHeader(http.StatusOK)
- w.Write([]byte("pong"))
-}
-
-// promHealthCheckHandler returns the health of Trickster
-// can't support multi-origin full proxy for path-based proxying
-func (t *TricksterHandler) promHealthCheckHandler(w http.ResponseWriter, r *http.Request) {
- level.Debug(t.Logger).Log(lfEvent, "promHealthCheckHandler", "path", r.URL.Path, "method", r.Method)
-
- // Check the labels path for Prometheus Origin Handler to satisfy health check
- path := prometheusAPIv1Path + mnLabels
-
- origin := t.getOrigin(r)
- originURL := origin.OriginURL + strings.Replace(path, "//", "/", 1)
- body, resp, _, err := t.getURL(origin, r.Method, originURL, r.URL.Query(), getProxyableClientHeaders(r))
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "error fetching data from origin Prometheus", lfDetail, err.Error())
- w.WriteHeader(http.StatusBadGateway)
- return
- }
-
- for k, v := range resp.Header {
- w.Header().Set(k, strings.Join(v, ","))
- }
-
- w.WriteHeader(resp.StatusCode)
- w.Write(body)
-}
-
-// promFullProxyHandler handles calls to non-api paths for single-origin configurations and multi-origin via param or hostname
-// can't support multi-origin full proxy for path-based proxying
-func (t *TricksterHandler) promFullProxyHandler(w http.ResponseWriter, r *http.Request) {
- level.Debug(t.Logger).Log(lfEvent, "promFullProxyHandler", "path", r.URL.Path, "method", r.Method)
-
- path := r.URL.Path
- vars := mux.Vars(r)
-
- // clear out the origin moniker from the front of the API path
- if originName, ok := vars["originMoniker"]; ok {
- if strings.HasPrefix(path, "/"+originName) {
- path = strings.Replace(path, "/"+originName, "", 1)
- }
- }
-
- origin := t.getOrigin(r)
- originURL := origin.OriginURL + strings.Replace(path, "//", "/", 1)
- body, resp, _, err := t.getURL(origin, r.Method, originURL, r.URL.Query(), getProxyableClientHeaders(r))
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "error fetching data from origin Prometheus", lfDetail, err.Error())
- w.WriteHeader(http.StatusBadGateway)
- return
- }
-
- for k, v := range resp.Header {
- w.Header().Set(k, strings.Join(v, ","))
- }
-
- writeResponse(w, body, resp)
-}
-
-// promQueryHandler handles calls to /query (for instantaneous values)
-func (t *TricksterHandler) promQueryHandler(w http.ResponseWriter, r *http.Request) {
- path := r.URL.Path
- vars := mux.Vars(r)
-
- // clear out the origin moniker from the front of the API path
- if originName, ok := vars["originMoniker"]; ok {
- if strings.HasPrefix(path, "/"+originName) {
- path = strings.Replace(path, "/"+originName, "", 1)
- }
- }
-
- originURL := t.getOrigin(r).OriginURL + strings.Replace(path, "//", "/", 1)
-
- // Get the params from the User request so we can inspect them and pass on to prometheus
- if err := r.ParseForm(); err != nil {
- level.Error(t.Logger).Log(lfEvent, "error parsing form", lfDetail, err.Error())
- w.WriteHeader(http.StatusBadRequest)
- return
- }
- params := r.Form
-
- body, resp, err := t.fetchPromQuery(originURL, params, r)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "error fetching data from origin Prometheus", lfDetail, err.Error())
- w.WriteHeader(http.StatusBadGateway)
- return
- }
-
- writeResponse(w, body, resp)
-}
-
-// promQueryRangeHandler handles calls to /query_range (requests for timeseries values)
-func (t *TricksterHandler) promQueryRangeHandler(w http.ResponseWriter, r *http.Request) {
- ctx, err := t.buildRequestContext(w, r)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "error building request context", lfDetail, err.Error())
- w.WriteHeader(http.StatusBadRequest)
- return
- }
-
- // This WaitGroup ensures that the server does not write the response until we are 100% done Trickstering the range request.
- // The responsders that fulfill client requests will mark the waitgroup done when the response is ready for delivery.
- ctx.WaitGroup.Add(1)
- if ctx.CacheLookupResult == crHit {
- t.respondToCacheHit(ctx)
- } else {
- t.queueRangeProxyRequest(ctx)
- }
-
- // Wait until the response is fulfilled before delivering.
- ctx.WaitGroup.Wait()
-}
-
-// End HTTP Handlers
-
-// Helper functions
-
-// defaultPrometheusMatrixEnvelope returns an empty envelope
-func defaultPrometheusMatrixEnvelope() PrometheusMatrixEnvelope {
- return PrometheusMatrixEnvelope{
- Data: PrometheusMatrixData{
- ResultType: rvMatrix,
- Result: make([]*model.SampleStream, 0),
- },
- }
-}
-
-// getProxyableClientHeaders returns any pertinent http headers from the client that we should pass through to the Origin when proxying
-func getProxyableClientHeaders(r *http.Request) http.Header {
- headers := http.Header{}
-
- // pass through Authorization Header
- if authorization, ok := r.Header[hnAuthorization]; ok {
- headers.Add(hnAuthorization, strings.Join(authorization, " "))
- }
-
- return headers
-}
-
-// getOrigin determines the origin server to service the request based on the Host header and url params
-func (t *TricksterHandler) getOrigin(r *http.Request) PrometheusOriginConfig {
- var originName string
- var ok bool
-
- vars := mux.Vars(r)
-
- // Check for the Origin Name URL Path
- if originName, ok = vars["originMoniker"]; !ok {
- // Check for the Origin Name URL Parmameter (origin=)
- if on, ok := r.URL.Query()[upOrigin]; ok {
- originName = on[1]
- } else {
- // Otherwise use the Host Header
- originName = r.Host
- }
- }
-
- // If we have matching origin in our Origins Map, return it.
- if p, ok := t.Config.Origins[originName]; ok {
- return p
- }
-
- // Otherwise, return the default origin if it is configured
- p, ok := t.Config.Origins["default"]
- if !ok {
- p = defaultOriginConfig()
- }
-
- if t.Config.DefaultOriginURL != "" {
- p.OriginURL = t.Config.DefaultOriginURL
- }
-
- return p
-}
-
-// setResponseHeaders adds any needed headers to the response object.
-// this should be called before the body is written
-func setResponseHeaders(w http.ResponseWriter, resp *http.Response) {
- // We're read only and a harmless API, so allow all CORS
- w.Header().Set(hnAllowOrigin, "*")
- // Set the Content-Type to what the response header is
- if contentType, ok := resp.Header["Content-Type"]; ok && len(contentType) > 0 {
- w.Header().Set(hnContentType, contentType[0])
- }
-}
-
-// getURL makes an HTTP request to the provided URL with the provided parameters and returns the response body
-func (t *TricksterHandler) getURL(o PrometheusOriginConfig, method string, uri string, params url.Values, headers http.Header) ([]byte, *http.Response, time.Duration, error) {
- if len(params) > 0 {
- uri += "?" + params.Encode()
- }
-
- parsedURL, err := url.Parse(uri)
- if err != nil {
- return nil, nil, 0, fmt.Errorf("error parsing URL %q: %v", uri, err)
- }
-
- startTime := time.Now()
- client := &http.Client{
- Timeout: time.Duration(o.TimeoutSecs * time.Second.Nanoseconds()),
- CheckRedirect: func(req *http.Request, via []*http.Request) error {
- return http.ErrUseLastResponse
- },
- }
-
- resp, err := client.Do(&http.Request{Method: method, URL: parsedURL})
- if err != nil {
- return nil, nil, 0, fmt.Errorf("error downloading URL %q: %v", uri, err)
- }
- defer resp.Body.Close()
-
- body, err := ioutil.ReadAll(resp.Body)
- if err != nil {
- return nil, nil, 0, fmt.Errorf("error reading body from HTTP response for URL %q: %v", uri, err)
- }
-
- if resp.StatusCode != http.StatusOK {
- // We don't want to return non-200 status codes as internal Go errors,
- // as we want to proxy those status codes all the way back to the user.
- level.Warn(t.Logger).Log(lfEvent, "error downloading URL", "url", uri, "status", resp.Status)
- return body, resp, 0, nil
- }
-
- duration := time.Since(startTime)
-
- level.Debug(t.Logger).Log(lfEvent, "prometheusOriginHttpRequest", "url", uri, "duration", duration)
-
- return body, resp, duration, nil
-}
-
-func (t *TricksterHandler) getVectorFromPrometheus(url string, params url.Values, r *http.Request) (PrometheusVectorEnvelope, []byte, *http.Response, error) {
- pe := PrometheusVectorEnvelope{}
-
- // Make the HTTP Request
- body, resp, err := t.fetchPromQuery(url, params, r)
- if err != nil {
- return pe, body, nil, fmt.Errorf("error fetching data from Prometheus: %v", err)
- }
- // Unmarshal the prometheus data into another PrometheusMatrixEnvelope
- err = json.Unmarshal(body, &pe)
- if err != nil {
- // If we get a scalar response, we just want to return the resp without an error
- // this will allow the upper layers to just use the raw response
- if pe.Data.ResultType != "scalar" {
- return pe, nil, nil, fmt.Errorf("Prometheus vector unmarshaling error for URL %q: %v", url, err)
- }
- }
-
- return pe, body, resp, nil
-}
-
-func (t *TricksterHandler) getMatrixFromPrometheus(url string, params url.Values, r *http.Request) (PrometheusMatrixEnvelope, []byte, *http.Response, time.Duration, error) {
- pe := PrometheusMatrixEnvelope{}
-
- // Make the HTTP Request - don't use fetchPromQuery here, that is for instantaneous only.
- body, resp, duration, err := t.getURL(t.getOrigin(r), r.Method, url, params, getProxyableClientHeaders(r))
- if err != nil {
- return pe, nil, nil, 0, err
- }
-
- if resp.StatusCode == http.StatusOK {
- // Unmarshal the prometheus data into another PrometheusMatrixEnvelope
- err := json.Unmarshal(body, &pe)
- if err != nil {
- return pe, nil, nil, 0, fmt.Errorf("Prometheus matrix unmarshaling error for URL %q: %v", url, err)
- }
- }
-
- return pe, body, resp, duration, nil
-}
-
-// fetchPromQuery checks for cached instantaneous value for the query and returns it if found,
-// otherwise proxies the request to the Prometheus origin and sets the cache with a low TTL
-// fetchPromQuery does not do any data marshalling
-func (t *TricksterHandler) fetchPromQuery(originURL string, params url.Values, r *http.Request) ([]byte, *http.Response, error) {
- var ttl int64 = 15
- var end int64
- var err error
-
- cacheKeyBase := originURL
- // if we have an authorization header, that should be part of the cache key to ensure only authorized users can access cached datasets
- if authorization, ok := r.Header[hnAuthorization]; ok {
- cacheKeyBase += strings.Join(authorization, " ")
- }
-
- if ts, ok := params[upTime]; ok {
- reqStart, err := parseTime(ts[0])
- if err != nil {
- return nil, nil, err
- }
- end = reqStart.Unix()
- if end <= (time.Now().Unix()-1800) && end%1800 == 0 {
- // the Time param is perfectly on the hour and not recent, this is unusual for random dashboard loads.
- // It might be some kind of a daily or hourly rollup. Let's cache it longer than 15s
- ttl = 1800
- }
- end = (end / 15) * 15
-
- params.Set(upTime, strconv.Itoa(int(end)))
- }
-
- cacheKey := deriveCacheKey(cacheKeyBase, params)
-
- var body []byte
- resp := &http.Response{}
- var duration time.Duration
-
- cacheResult := crKeyMiss
-
- // check for it in the cache
- cachedBody, err := t.Cacher.Retrieve(cacheKey)
- if err != nil {
- // Cache Miss, we need to get it from prometheus
- body, resp, duration, err = t.getURL(t.getOrigin(r), r.Method, originURL, params, getProxyableClientHeaders(r))
- if err != nil {
- return nil, nil, err
- }
-
- t.Metrics.ProxyRequestDuration.WithLabelValues(originURL, otPrometheus, mnQuery, crKeyMiss, strconv.Itoa(resp.StatusCode)).Observe(duration.Seconds())
- t.Cacher.Store(cacheKey, string(body), ttl)
- } else {
- // Cache hit, return the data set
- body = []byte(cachedBody)
- cacheResult = crHit
- resp.StatusCode = http.StatusOK
- }
-
- t.Metrics.CacheRequestStatus.WithLabelValues(originURL, otPrometheus, mnQuery, cacheResult, strconv.Itoa(resp.StatusCode)).Inc()
-
- return body, resp, nil
-}
-
-// buildRequestContext Creates a ClientRequestContext based on the incoming client request
-func (t *TricksterHandler) buildRequestContext(w http.ResponseWriter, r *http.Request) (*ClientRequestContext, error) {
- var err error
-
- ctx := &ClientRequestContext{
- Request: r,
- Writer: w,
- Origin: t.getOrigin(r),
- Time: time.Now().Unix(),
- }
-
- ctx.Origin.OriginURL += strings.Replace(ctx.Origin.APIPath+"/", "//", "/", 1)
-
- // Get the params from the User request so we can inspect them and pass on to prometheus
- if err := r.ParseForm(); err != nil {
- return nil, errors.Wrap(err, "unable to parse form")
- }
- ctx.RequestParams = r.Form
-
- // Validate and parse the step value from the user request URL params.
- if len(ctx.RequestParams[upStep]) == 0 {
- return nil, fmt.Errorf("missing step parameter")
- }
- ctx.StepParam = ctx.RequestParams[upStep][0]
- step, err := parseDuration(ctx.StepParam)
- if err != nil {
- return nil, errors.Wrap(err, fmt.Sprintf("failed to parse parameter %q with value %q", upStep, ctx.StepParam))
- }
- if step <= 0 {
- return nil, fmt.Errorf("step parameter %v <= 0, has to be positive", step)
- }
- ctx.StepMS = int64(step.Seconds() * 1000)
-
- cacheKeyBase := ctx.Origin.OriginURL + ctx.StepParam
- // if we have an authorization header, that should be part of the cache key to ensure only authorized users can access cached datasets
- if authorization, ok := r.Header[hnAuthorization]; ok {
- cacheKeyBase += strings.Join(authorization, " ")
- }
-
- // Derive a hashed cacheKey for the query where we will get and set the result set
- // inclusion of the step ensures that datasets with different resolutions are not written to the same key.
- ctx.CacheKey = deriveCacheKey(cacheKeyBase, ctx.RequestParams)
-
- // We will look for a Cache-Control: No-Cache request header and,
- // if present, bypass the cache for a fresh full query from prometheus.
- // Any user can trigger w/ hard reload (ctrl/cmd+shift+r) to clear out cache-related anomalies
- noCache := false
- if ctx.Origin.IgnoreNoCacheHeader == false && (strings.ToLower(r.Header.Get(hnCacheControl)) == hvNoCache) {
- noCache = true
- }
-
- // get the browser-requested start/end times, so we can determine what part of the range is not in the cache
- if len(ctx.RequestParams[upStart]) == 0 {
- return nil, fmt.Errorf("missing start time parameter")
- }
-
- reqStart, err := parseTime(ctx.RequestParams[upStart][0])
- if err != nil {
- return nil, errors.Wrap(err, fmt.Sprintf("failed to parse parameter %q with value %q", upStart, ctx.RequestParams[upStart][0]))
- }
-
- if len(ctx.RequestParams[upEnd]) == 0 {
- return nil, fmt.Errorf("missing end time parameter")
- }
-
- reqEnd, err := parseTime(ctx.RequestParams[upEnd][0])
- if err != nil {
- return nil, errors.Wrap(err, fmt.Sprintf("failed to parse parameter %q with value %q", upEnd, ctx.RequestParams[upEnd][0]))
- }
-
- ctx.RequestExtents.Start, ctx.RequestExtents.End, err = alignStepBoundaries(reqStart.Unix()*1000, reqEnd.Unix()*1000, ctx.StepMS, ctx.Time)
- if err != nil {
- return nil, errors.Wrap(err, "error aligning step boundary")
- }
- // setup some variables to determine and track the status of the query vs what's in the cache
- ctx.Matrix = defaultPrometheusMatrixEnvelope()
- ctx.CacheLookupResult = crKeyMiss
-
- // parameters for filling gap on the upper bound
- ctx.OriginUpperExtents.Start = ctx.RequestExtents.Start
- ctx.OriginUpperExtents.End = ctx.RequestExtents.End
-
- // If the entire request is outside of the MaxValueAgeSecs, then lets not look in the
- // cache, we won't have it
- if (ctx.Time*1000 - ctx.RequestExtents.End) > ctx.Origin.MaxValueAgeSecs*1000 {
- ctx.CacheLookupResult = crRangeMiss
- return ctx, nil
- }
-
- // Get the cached result set if present
- cachedBody, err := t.Cacher.Retrieve(ctx.CacheKey)
-
- if err != nil || noCache {
- // Cache Miss, Get the whole blob from Prometheus.
- // Pass on the browser-requested start/end parameters to our Prom Query
- if noCache {
- ctx.CacheLookupResult = crPurge
- }
- } else {
- // We had a Redis Key Hit for the hashed query key, but we may not have all points requested by browser
- // So we can have a Range Miss, Partial Hit, Full Hit when comparing cached range to what the client requested.
- // So let's find out what we are missing (if anything) and fetch what we don't have
-
- // See if cache data is compressed by looking for the first character to be "{":, with which the uncompressed JSON would start
- // We do this instead of checking the Compression config bit because if someone turns compression on or off when using filesystem or redis cache,
- // we will have no idea if what is already in the cache was compressed or not based on previous settings
- cb := []byte(cachedBody)
- if cb[0] != 123 {
- // Not a JSON object, try decompressing
- level.Debug(t.Logger).Log("event", "Decompressing Cached Data", "cacheKey", ctx.CacheKey)
- cb, err = snappy.Decode(nil, cb)
- if err == nil {
- cachedBody = string(cb)
- }
- }
-
- // Marshall the cache payload into a PrometheusMatrixEnvelope struct
- err = json.Unmarshal([]byte(cachedBody), &ctx.Matrix)
- // If there is an error unmarshaling the cache we should treat it as a cache miss
- // and re-fetch from origin
- if err != nil {
- ctx.CacheLookupResult = crRangeMiss
- return ctx, nil
- }
-
- // Get the Extents of the data in the cache
- ce := ctx.Matrix.getExtents()
-
- extent := "none"
-
- // Figure out our Deltas
- if ce.End == 0 || ce.Start == 0 {
- // Something went wrong fetching extents
- ctx.CacheLookupResult = crRangeMiss
- } else if ctx.RequestExtents.Start >= ce.Start && ctx.RequestExtents.End <= ce.End {
- // Full cache hit, no need to refresh dataset.
- // Everything we are requesting is already in cache
- ctx.CacheLookupResult = crHit
- ctx.OriginUpperExtents.Start = 0
- ctx.OriginUpperExtents.End = 0
- } else if ctx.RequestExtents.Start < ce.Start && ctx.RequestExtents.End > ce.End {
- // Partial Cache hit on both ends.
- ctx.CacheLookupResult = crPartialHit
- ctx.OriginUpperExtents.Start = ce.End + ctx.StepMS
- ctx.OriginUpperExtents.End = ctx.RequestExtents.End
- ctx.OriginLowerExtents.Start = ((ctx.RequestExtents.Start / ctx.StepMS) * ctx.StepMS)
- ctx.OriginLowerExtents.End = ce.Start
- extent = "both"
- } else if ctx.RequestExtents.Start > ce.End {
- // Range Miss on the Upper Extent of Cache. We will fill from where our cached data stops to the requested end
- ctx.CacheLookupResult = crRangeMiss
- ctx.OriginUpperExtents.Start = ce.End + ctx.StepMS
- extent = "upper"
- } else if ctx.RequestExtents.End > ce.End {
- // Partial Cache Hit, Missing the Upper Extent
- ctx.CacheLookupResult = crPartialHit
- ctx.OriginUpperExtents.Start = ce.End + ctx.StepMS
- extent = "upper"
- } else if ctx.RequestExtents.End < ce.Start {
- // Range Miss on the Lower Extent of Cache. We will fill from the requested start up to where our cached data stops
- ctx.CacheLookupResult = crRangeMiss
- ctx.OriginLowerExtents.Start = ((ctx.RequestExtents.Start / ctx.StepMS) * ctx.StepMS)
- ctx.OriginLowerExtents.End = ce.Start - ctx.StepMS
- ctx.OriginUpperExtents.Start = 0
- ctx.OriginUpperExtents.End = 0
- extent = "lower"
- } else if ctx.RequestExtents.Start < ce.Start {
- // Partial Cache Hit, Missing Lower Extent
- ctx.CacheLookupResult = crPartialHit
- ctx.OriginLowerExtents.Start = ((ctx.RequestExtents.Start / ctx.StepMS) * ctx.StepMS)
- ctx.OriginLowerExtents.End = ce.Start - ctx.StepMS
- ctx.OriginUpperExtents.Start = 0
- ctx.OriginUpperExtents.End = 0
- extent = "upper"
- } else {
- panic(fmt.Sprintf("Reaching this final clause should be impossible. Yikes! reqStart=%d, reqEnd=%d, ce.Start=%d, ce.End=%d", ctx.RequestExtents.Start, ctx.RequestExtents.End, ce.Start, ce.End))
- }
-
- level.Debug(t.Logger).Log(lfEvent, "deltaRoutineCompleted", "CacheLookupResult", ctx.CacheLookupResult, lfCacheKey, ctx.CacheKey,
- "cacheStart", ce.Start, "cacheEnd", ce.End, "reqStart", ctx.RequestExtents.Start, "reqEnd", ctx.RequestExtents.End,
- "OriginLowerExtents.Start", ctx.OriginLowerExtents.Start, "OriginLowerExtents.End", ctx.OriginLowerExtents.End,
- "OriginUpperExtents.Start", ctx.OriginUpperExtents.Start, "OriginUpperExtents.End", ctx.OriginUpperExtents.End, "extent", extent)
- }
-
- return ctx, nil
-}
-
-func (t *TricksterHandler) respondToCacheHit(ctx *ClientRequestContext) {
- defer ctx.WaitGroup.Done()
- t.Metrics.CacheRequestStatus.WithLabelValues(ctx.Origin.OriginURL, otPrometheus, mnQueryRange, ctx.CacheLookupResult, "200").Inc()
-
- // Do the extraction of the range the user requested from the fully cached dataset, if needed.
- ctx.Matrix.cropToRange(ctx.RequestExtents.Start, ctx.RequestExtents.End+ctx.StepMS)
-
- r := &http.Response{}
-
- // If Fast Forward is enabled and the request is a real-time request, go get that data
- if !ctx.Origin.FastForwardDisable && !(ctx.RequestExtents.End < (ctx.Time*1000)-ctx.StepMS) {
- // Query the latest points if Fast Forward is enabled
- queryURL := ctx.Origin.OriginURL + mnQuery
- originParams := url.Values{}
- // Add the prometheus query params from the user urlparams to the origin request
- passthroughParam(upQuery, ctx.RequestParams, originParams, nil)
- passthroughParam(upTimeout, ctx.RequestParams, originParams, nil)
- passthroughParam(upTime, ctx.RequestParams, originParams, nil)
- ffd, _, resp, err := t.getVectorFromPrometheus(queryURL, originParams, ctx.Request)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "error fetching data from origin Prometheus", lfDetail, err.Error())
- ctx.Writer.WriteHeader(http.StatusBadGateway)
- return
- }
- r = resp
- if resp.StatusCode == http.StatusOK && ffd.Status == rvSuccess {
- ctx.Matrix = t.mergeVector(ctx.Matrix, ffd)
- }
- }
-
- // Marshal the Envelope back to a json object for User Response)
- body, err := json.Marshal(ctx.Matrix)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "prometheus matrix marshaling error", lfDetail, err.Error())
- ctx.Writer.WriteHeader(http.StatusInternalServerError)
- return
- }
-
- writeResponse(ctx.Writer, body, r)
-}
-
-func writeResponse(w http.ResponseWriter, body []byte, resp *http.Response) {
- // Now we need to respond to the user request with the dataset
- setResponseHeaders(w, resp)
-
- if resp.StatusCode == 0 {
- resp.StatusCode = http.StatusOK
- }
-
- w.WriteHeader(resp.StatusCode)
- w.Write(body)
-}
-
-func (t *TricksterHandler) queueRangeProxyRequest(ctx *ClientRequestContext) {
- t.ChannelCreateMtx.Lock()
- ch, ok := t.ResponseChannels[ctx.CacheKey]
- if !ok {
- level.Info(t.Logger).Log(lfEvent, "starting originRangeProxyHandler", lfCacheKey, ctx.CacheKey)
- ch = make(chan *ClientRequestContext, 100)
- t.ResponseChannels[ctx.CacheKey] = ch
- go t.originRangeProxyHandler(ctx.CacheKey, ch)
- }
- t.ChannelCreateMtx.Unlock()
-
- ch <- ctx
-}
-
-func (t *TricksterHandler) originRangeProxyHandler(cacheKey string, originRangeRequests <-chan *ClientRequestContext) {
- // Close handler goroutine if its request channel is empty.
- go func() {
- for {
- time.Sleep(10 * time.Second)
-
- t.ChannelCreateMtx.Lock()
-
- if len(originRangeRequests) == 0 {
- if _, ok := t.ResponseChannels[cacheKey]; ok {
- close(t.ResponseChannels[cacheKey])
- delete(t.ResponseChannels, cacheKey)
- t.ChannelCreateMtx.Unlock()
- return
- }
- }
-
- t.ChannelCreateMtx.Unlock()
- }
- }()
-
- for r := range originRangeRequests {
- // get the cache data for this request again, in case anything about the record has changed
- // between the time we queued the request and the time it was consumed from the channel
- ctx, err := t.buildRequestContext(r.Writer, r.Request)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "error building request context", lfDetail, err.Error())
- r.Writer.WriteHeader(http.StatusBadRequest)
- r.WaitGroup.Done()
- continue
- }
-
- // The cache miss became a cache hit between the time it was queued and processed.
- if ctx.CacheLookupResult == crHit {
- level.Debug(t.Logger).Log(lfEvent, "delayedCacheHit", lfDetail, "cache was populated with needed data by another proxy request while this one was queued.")
- // Lay the newly-retreived data into the original origin range request so it can fully service the client
- r.Matrix = ctx.Matrix
- // And change the lookup result to a hit.
- r.CacheLookupResult = crHit
- // Respond with the modified original request object so the right WaitGroup is marked as Done()
- t.respondToCacheHit(r)
- } else {
-
- // Now we know if we need to make any calls to the Origin, lets set those up
- upperDeltaData := PrometheusMatrixEnvelope{}
- lowerDeltaData := PrometheusMatrixEnvelope{}
- fastForwardData := PrometheusVectorEnvelope{}
-
- var wg sync.WaitGroup
-
- var m sync.Mutex // Protects originErr and resp below.
- var originErr error
- var errorBody []byte
- resp := &http.Response{}
-
- if ctx.OriginLowerExtents.Start > 0 && ctx.OriginLowerExtents.End > 0 {
- wg.Add(1)
- go func() {
- defer wg.Done()
-
- queryURL := ctx.Origin.OriginURL + mnQueryRange
- originParams := url.Values{}
- // Add the prometheus query params from the user urlparams to the origin request
- passthroughParam(upQuery, ctx.RequestParams, originParams, nil)
- passthroughParam(upTimeout, ctx.RequestParams, originParams, nil)
- originParams.Add(upStep, ctx.StepParam)
- originParams.Add(upStart, strconv.FormatInt(ctx.OriginLowerExtents.Start/1000, 10))
- originParams.Add(upEnd, strconv.FormatInt(ctx.OriginLowerExtents.End/1000, 10))
- ldd, b, r, duration, err := t.getMatrixFromPrometheus(queryURL, originParams, r.Request)
-
- if err != nil {
- m.Lock()
- originErr = err
- m.Unlock()
- return
- }
-
- m.Lock()
- if resp.StatusCode == 0 || r.StatusCode != http.StatusOK {
- if r.StatusCode != http.StatusOK {
- errorBody = b
- }
- resp = r
- }
- m.Unlock()
-
- if r.StatusCode == http.StatusOK && ldd.Status == rvSuccess {
- lowerDeltaData = ldd
- t.Metrics.ProxyRequestDuration.WithLabelValues(ctx.Origin.OriginURL, otPrometheus,
- mnQueryRange, ctx.CacheLookupResult, strconv.Itoa(r.StatusCode)).Observe(duration.Seconds())
- }
- }()
- }
-
- if ctx.OriginUpperExtents.Start > 0 && ctx.OriginUpperExtents.End > 0 {
- wg.Add(1)
- go func() {
- defer wg.Done()
-
- queryURL := ctx.Origin.OriginURL + mnQueryRange
- originParams := url.Values{}
- // Add the prometheus query params from the user urlparams to the origin request
- passthroughParam(upQuery, ctx.RequestParams, originParams, nil)
- passthroughParam(upTimeout, ctx.RequestParams, originParams, nil)
- originParams.Add(upStep, ctx.StepParam)
- originParams.Add(upStart, strconv.FormatInt(ctx.OriginUpperExtents.Start/1000, 10))
- originParams.Add(upEnd, strconv.FormatInt(ctx.OriginUpperExtents.End/1000, 10))
- udd, b, r, duration, err := t.getMatrixFromPrometheus(queryURL, originParams, r.Request)
-
- if err != nil {
- m.Lock()
- originErr = err
- m.Unlock()
- return
- }
-
- m.Lock()
- if resp.StatusCode == 0 || r.StatusCode != http.StatusOK {
- if r.StatusCode != http.StatusOK {
- errorBody = b
- }
- resp = r
- }
- m.Unlock()
-
- if r != nil && r.StatusCode == http.StatusOK && udd.Status == rvSuccess {
- upperDeltaData = udd
- t.Metrics.ProxyRequestDuration.WithLabelValues(ctx.Origin.OriginURL, otPrometheus,
- mnQueryRange, ctx.CacheLookupResult, strconv.Itoa(r.StatusCode)).Observe(duration.Seconds())
- }
- }()
- }
-
- if !ctx.Origin.FastForwardDisable && !(ctx.RequestExtents.End < ctx.Time*1000-ctx.StepMS) {
- wg.Add(1)
- go func() {
- defer wg.Done()
-
- // Query the latest points if Fast Forward is enabled
- queryURL := ctx.Origin.OriginURL + mnQuery
- originParams := url.Values{}
- // Add the prometheus query params from the user urlparams to the origin request
- passthroughParam(upQuery, ctx.RequestParams, originParams, nil)
- passthroughParam(upTimeout, ctx.RequestParams, originParams, nil)
- passthroughParam(upTime, ctx.RequestParams, originParams, nil)
- ffd, b, r, err := t.getVectorFromPrometheus(queryURL, originParams, r.Request)
-
- if err != nil {
- m.Lock()
- originErr = err
- m.Unlock()
- return
- }
-
- m.Lock()
- if resp.StatusCode == 0 || r.StatusCode != http.StatusOK {
- if r.StatusCode != http.StatusOK {
- errorBody = b
- }
- resp = r
- }
- m.Unlock()
-
- if r != nil && r.StatusCode == http.StatusOK && ffd.Status == rvSuccess {
- fastForwardData = ffd
- }
- }()
- }
-
- wg.Wait()
-
- if originErr != nil {
- level.Error(t.Logger).Log(lfEvent, "error fetching data from origin Prometheus", lfDetail, originErr.Error())
- r.Writer.WriteHeader(http.StatusBadGateway)
- r.WaitGroup.Done()
- continue
- }
-
- t.Metrics.CacheRequestStatus.WithLabelValues(ctx.Origin.OriginURL, otPrometheus, mnQueryRange, ctx.CacheLookupResult, strconv.Itoa(resp.StatusCode)).Inc()
-
- uncachedElementCnt := int64(0)
-
- if lowerDeltaData.Status == rvSuccess {
- uncachedElementCnt += lowerDeltaData.getValueCount()
- ctx.Matrix = t.mergeMatrix(ctx.Matrix, lowerDeltaData)
- }
-
- if upperDeltaData.Status == rvSuccess {
- uncachedElementCnt += upperDeltaData.getValueCount()
- ctx.Matrix = t.mergeMatrix(upperDeltaData, ctx.Matrix)
- }
-
- // If the request is entirely outside of the cache window, we don't want to cache it
- // otherwise we actually *clear* the cache of any data it has in it!
- skipCache := (ctx.Time*1000 - ctx.RequestExtents.End) > ctx.Origin.MaxValueAgeSecs*1000
-
- // If it's not a full cache hit, we want to write this back to the cache
- if ctx.CacheLookupResult != crHit && !skipCache {
- cacheMatrix := ctx.Matrix.copy()
-
- // Prune any old points based on retention policy
- cacheMatrix.cropToRange(int64(ctx.Time-ctx.Origin.MaxValueAgeSecs)*1000, 0)
-
- if ctx.Origin.NoCacheLastDataSecs != 0 {
- cacheMatrix.cropToRange(0, int64(ctx.Time-ctx.Origin.NoCacheLastDataSecs)*1000)
- }
-
- // Marshal the Envelope back to a json object for Cache Storage
- cacheBody, err := json.Marshal(cacheMatrix)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "prometheus matrix marshaling error", lfDetail, err.Error())
- r.Writer.WriteHeader(http.StatusInternalServerError)
- r.WaitGroup.Done()
- continue
- }
-
- if t.Config.Caching.Compression {
- level.Debug(t.Logger).Log("event", "Compressing Cached Data", "cacheKey", ctx.CacheKey)
- cacheBody = snappy.Encode(nil, cacheBody)
- }
-
- // Set the Cache Key with the merged dataset
- t.Cacher.Store(cacheKey, string(cacheBody), t.Config.Caching.RecordTTLSecs)
- level.Debug(t.Logger).Log(lfEvent, "setCacheRecord", lfCacheKey, cacheKey, "ttl", t.Config.Caching.RecordTTLSecs)
- }
-
- //Do the extraction of the range the user requested, if needed.
- // The only time it may not be needed is if the result was a Key Miss (so the dataset we have is exactly what the user asked for)
- // I add one more step on the end of the request to ensure we catch the fast forward data
- if ctx.CacheLookupResult != crKeyMiss {
- ctx.Matrix.cropToRange(ctx.RequestExtents.Start, ctx.RequestExtents.End+ctx.StepMS)
- }
-
- allElementCnt := ctx.Matrix.getValueCount()
- cachedElementCnt := allElementCnt - uncachedElementCnt
-
- if uncachedElementCnt > 0 {
- t.Metrics.CacheRequestElements.WithLabelValues(ctx.Origin.OriginURL, otPrometheus, "uncached").Add(float64(uncachedElementCnt))
- }
-
- if cachedElementCnt > 0 {
- t.Metrics.CacheRequestElements.WithLabelValues(ctx.Origin.OriginURL, otPrometheus, "cached").Add(float64(cachedElementCnt))
- }
-
- // Stictch in Fast Forward Data
- if fastForwardData.Status == rvSuccess {
- ctx.Matrix = t.mergeVector(ctx.Matrix, fastForwardData)
- }
-
- // Marshal the Envelope back to a json object for User Response)
- body, err := json.Marshal(ctx.Matrix)
- if err != nil {
- level.Error(t.Logger).Log(lfEvent, "prometheus matrix marshaling error", lfDetail, err.Error())
- r.Writer.WriteHeader(http.StatusInternalServerError)
- r.WaitGroup.Done()
- continue
- }
-
- if resp.StatusCode != http.StatusOK {
- writeResponse(r.Writer, errorBody, resp)
- } else {
- writeResponse(r.Writer, body, resp)
- }
- r.WaitGroup.Done()
- }
- // Explicitly release the request context so that the underlying memory can be
- // freed before the next request is received via the channel, which overwrites "r".
- r = nil
- }
-}
-
-func alignStepBoundaries(start int64, end int64, stepMS int64, now int64) (int64, int64, error) {
- // Don't query beyond Time.Now() or charts will have weird data on the far right
- if end > now*1000 {
- end = now * 1000
- }
-
- // In case the user had the start/end parameters reversed chronologically, lets return an error
- if start > end {
- return 0, 0, fmt.Errorf("start is after end")
- }
-
- // Failsafe to 60s if something inexplicably happened to the step param
- if stepMS <= 0 {
- return 0, 0, fmt.Errorf("step must be > 0")
- }
-
- // Align start/end to step boundaries
- start = (start / stepMS) * stepMS
- end = ((end / stepMS) * stepMS)
-
- return start, end, nil
-}
-
-func (pe PrometheusMatrixEnvelope) getValueCount() int64 {
- i := int64(0)
- for j := range pe.Data.Result {
- i += int64(len(pe.Data.Result[j].Values))
- }
- return i
-}
-
-// mergeVector merges the passed PrometheusVectorEnvelope object with the calling PrometheusVectorEnvelope object
-func (t *TricksterHandler) mergeVector(pe PrometheusMatrixEnvelope, pv PrometheusVectorEnvelope) PrometheusMatrixEnvelope {
- if len(pv.Data.Result) == 0 {
- level.Debug(t.Logger).Log(lfEvent, "mergeVectorPrematureExit")
- return pe
- }
-
- for i := range pv.Data.Result {
- result2 := pv.Data.Result[i]
- for j := range pe.Data.Result {
- result1 := pe.Data.Result[j]
- if result2.Metric.Equal(result1.Metric) {
- if result2.Timestamp > result1.Values[len(result1.Values)-1].Timestamp {
- pe.Data.Result[j].Values = append(pe.Data.Result[j].Values, model.SamplePair{
- Timestamp: model.Time((int64(result2.Timestamp) / 1000) * 1000),
- Value: result2.Value,
- })
- }
- }
- }
- }
-
- return pe
-}
-
-// mergeMatrix merges the passed PrometheusMatrixEnvelope object with the calling PrometheusMatrixEnvelope object
-func (t *TricksterHandler) mergeMatrix(pe PrometheusMatrixEnvelope, pe2 PrometheusMatrixEnvelope) PrometheusMatrixEnvelope {
- if pe.Status != rvSuccess {
- pe = pe2
- return pe2
- } else if pe2.Status != rvSuccess {
- return pe
- }
-
- for i := range pe2.Data.Result {
- metricSetFound := false
- result2 := pe2.Data.Result[i]
- METRIC_MERGE:
- for j := range pe.Data.Result {
- result1 := pe.Data.Result[j]
- if result2.Metric.Equal(result1.Metric) {
- metricSetFound = true
- // Ensure that we don't duplicate datapoints or put points out-of-order
- // This method assumes that `pe2` is "before" `pe`, we need to actually
- // check and enforce that assumption
- first := result1.Values[0]
- for x := len(result2.Values) - 1; x >= 0; x-- {
- v := result2.Values[x]
- if v.Timestamp < first.Timestamp {
- result1.Values = append(result2.Values[:x+1], result1.Values...)
- break METRIC_MERGE
- }
- }
- break METRIC_MERGE
- }
- }
-
- if !metricSetFound {
- level.Debug(t.Logger).Log(lfEvent, "MergeMatrixEnvelopeNewMetric", lfDetail, "Did not find mergeable metric set in cache", "metricFingerprint", result2.Metric.Fingerprint())
- // Couldn't find metrics with that name in the existing resultset, so this must
- // be new for this poll. That's fine, just add it outright instead of merging.
- pe.Data.Result = append(pe.Data.Result, result2)
- }
- }
-
- return pe
-}
-
-// cropToRange crops the datasets in a given PrometheusMatrixEnvelope down to the provided start and end times
-func (pe *PrometheusMatrixEnvelope) cropToRange(start int64, end int64) {
- seriesToRemove := make([]int, 0)
-
- // iterate through each metric series in the result
- for i := range pe.Data.Result {
- if start > 0 {
- // Now we First determine the correct start index for each series in the Matrix
- // iterate through each value in the given metric series
- for j := range pe.Data.Result[i].Values {
- // If the timestamp for this data point is at or after the client requested start time,
- // update the slice and break the loop.
- ts := int64(pe.Data.Result[i].Values[j].Timestamp)
- if ts >= start {
- pe.Data.Result[i].Values = pe.Data.Result[i].Values[j:]
- break
- }
- }
-
- if len(pe.Data.Result[i].Values) == 0 || int64(pe.Data.Result[i].Values[len(pe.Data.Result[i].Values)-1].Timestamp) < start {
- seriesToRemove = append(seriesToRemove, i)
- }
- }
-
- if end > 0 {
- // Then we determine the correct end index for each series in the Matrix
- // iterate *backwards* through each value in the given metric series
- for j := len(pe.Data.Result[i].Values) - 1; j >= 0; j-- {
- // If the timestamp of this metric is at or after the client requested start time,
- // update the offset and break.
- ts := int64(pe.Data.Result[i].Values[j].Timestamp)
- if ts <= end {
- pe.Data.Result[i].Values = pe.Data.Result[i].Values[:j+1]
- break
- }
- }
-
- if len(pe.Data.Result[i].Values) == 0 || int64(pe.Data.Result[i].Values[0].Timestamp) > end {
- if len(seriesToRemove) == 0 || seriesToRemove[len(seriesToRemove)-1] != i {
- seriesToRemove = append(seriesToRemove, i)
- }
- }
- }
- }
-
- for i := len(seriesToRemove) - 1; i >= 0; i-- {
- toRemove := seriesToRemove[i]
- pe.Data.Result = append(pe.Data.Result[:toRemove], pe.Data.Result[toRemove+1:]...)
- }
-}
-
-// getCacheExtents returns the timestamps of the oldest and newest cached data points for the given query.
-func (pe PrometheusMatrixEnvelope) getExtents() MatrixExtents {
- r := pe.Data.Result
-
- var oldest int64
- var newest int64
-
- for series := range r {
- if len(r[series].Values) > 0 {
- // Update Oldest Value
- ts := int64(r[series].Values[0].Timestamp)
- if oldest == 0 || ts < oldest {
- oldest = ts
- }
-
- // Update Newest Value
- ts = int64(r[series].Values[len(r[series].Values)-1].Timestamp)
- if newest == 0 || ts > newest {
- newest = ts
- }
- }
- }
-
- return MatrixExtents{Start: oldest, End: newest}
-}
-
-// copy return a deep copy of PrometheusMatrixEnvelope.
-func (pe PrometheusMatrixEnvelope) copy() PrometheusMatrixEnvelope {
- resPe := PrometheusMatrixEnvelope{
- Status: pe.Status,
- Data: PrometheusMatrixData{
- ResultType: pe.Data.ResultType,
- Result: make([]*model.SampleStream, len(pe.Data.Result)),
- },
- }
- for index := range pe.Data.Result {
- resSampleSteam := *pe.Data.Result[index]
- resPe.Data.Result[index] = &resSampleSteam
- }
- return resPe
-}
-
-// passthroughParam passes the parameter with paramName, if present in the requestParams, on to the proxyParams collection
-func passthroughParam(paramName string, requestParams url.Values, proxyParams url.Values, filterFunc func(string) string) {
- if value, ok := requestParams[paramName]; ok {
- if filterFunc != nil {
- value[0] = filterFunc(value[0])
- }
- proxyParams.Add(paramName, value[0])
- }
-}
-
-// md5sum returns the calculated hex string version of the md5 checksum for the input string
-func md5sum(input string) string {
- return fmt.Sprintf("%x", md5.Sum([]byte(input)))
-}
-
-// deriveCacheKey calculates a query-specific keyname based on the prometheus query in the user request
-func deriveCacheKey(prefix string, params url.Values) string {
- k := ""
- // if we have a prefix, set it up
- if len(prefix) > 0 {
- k = md5sum(prefix)
- }
-
- if query, ok := params[upQuery]; ok {
- k += "." + md5sum(query[0])
- }
-
- if t, ok := params[upTime]; ok {
- k += "." + md5sum(t[0])
- }
-
- return k
-}
-
-var reRelativeTime = regexp.MustCompile(`([0-9]+)([mshdw])`)
-
-// parseTime converts a query time URL parameter to time.Time.
-// Copied from https://github.com/prometheus/prometheus/blob/v2.2.1/web/api/v1/api.go#L798-L807
-func parseTime(s string) (time.Time, error) {
- if t, err := strconv.ParseFloat(s, 64); err == nil {
- s, ns := math.Modf(t)
- return time.Unix(int64(s), int64(ns*float64(time.Second))), nil
- }
- if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
- return t, nil
- }
- return time.Time{}, fmt.Errorf("cannot parse %q to a valid timestamp", s)
-}
-
-// parseDuration converts a duration URL parameter to time.Duration.
-// Copied from https://github.com/prometheus/prometheus/blob/v2.2.1/web/api/v1/api.go#L809-L821
-func parseDuration(s string) (time.Duration, error) {
- if d, err := strconv.ParseFloat(s, 64); err == nil {
- ts := d * float64(time.Second)
- if ts > float64(math.MaxInt64) || ts < float64(math.MinInt64) {
- return 0, fmt.Errorf("cannot parse %q to a valid duration. It overflows int64", s)
- }
- return time.Duration(ts), nil
- }
- if d, err := model.ParseDuration(s); err == nil {
- return time.Duration(d), nil
- }
- return 0, fmt.Errorf("cannot parse %q to a valid duration", s)
-}
diff --git a/handlers_test.go b/handlers_test.go
deleted file mode 100644
index 9ec2b5dda..000000000
--- a/handlers_test.go
+++ /dev/null
@@ -1,752 +0,0 @@
-/**
-* Copyright 2018 Comcast Cable Communications Management, LLC
-* Licensed under the Apache License, Version 2.0 (the "License");
-* you may not use this file except in compliance with the License.
-* You may obtain a copy of the License at
-* http://www.apache.org/licenses/LICENSE-2.0
-* Unless required by applicable law or agreed to in writing, software
-* distributed under the License is distributed on an "AS IS" BASIS,
-* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-* See the License for the specific language governing permissions and
-* limitations under the License.
- */
-
-package main
-
-import (
- "bytes"
- "encoding/json"
- "fmt"
- "io/ioutil"
- "net/http"
- "net/http/httptest"
- "net/url"
- "reflect"
- "strconv"
- "testing"
-
- "github.com/go-kit/kit/log"
- "github.com/prometheus/common/model"
-)
-
-const (
- nonexistantOrigin = "http://nonexistent-origin:54321"
- exampleQuery = "/api/v1/query?query=up&time=2015-07-01T20:11:15.781Z"
- exampleRangeQuery = "/api/v1/query_range?query=up&start=2015-07-01T20:10:30.781Z&end=2015-07-01T20:11:00.781Z&step=15"
- exampleRangeQuery_query = "up"
- exampleRangeQuery_start = "2015-07-01T20:10:30.781Z"
- exampleRangeQuery_end = "2015-07-01T20:11:00.781Z"
- exampleRangeQuery_step = "15"
-
- // this example should have 2 data points later than those in exampleRangeResponse
- exampleResponse = `{
- "status" : "success",
- "data" : {
- "resultType" : "vector",
- "result" : [
- {
- "metric" : {
- "__name__" : "up",
- "job" : "prometheus",
- "instance" : "localhost:9090"
- },
- "value": [ 1435781475.781, "1" ]
- },
- {
- "metric" : {
- "__name__" : "up",
- "job" : "node",
- "instance" : "localhost:9091"
- },
- "value" : [ 1435781475.781, "0" ]
- }
- ]
- }
-}`
-
- // this example should have 6 data points
- // NOTE: Times in this response should end with '.781' not '.000'. Had
- // to truncate due to how extents are measured in TricksterHandler.
- exampleRangeResponse = `{
- "status" : "success",
- "data" : {
- "resultType" : "matrix",
- "result" : [
- {
- "metric" : {
- "__name__" : "up",
- "job" : "prometheus",
- "instance" : "localhost:9090"
- },
- "values" : [
- [ 1435781430.000, "1" ],
- [ 1435781445.000, "1" ],
- [ 1435781460.000, "1" ]
- ]
- },
- {
- "metric" : {
- "__name__" : "up",
- "job" : "node",
- "instance" : "localhost:9091"
- },
- "values" : [
- [ 1435781430.000, "0" ],
- [ 1435781445.000, "0" ],
- [ 1435781460.000, "1" ]
- ]
- }
- ]
- }
-}`
-)
-
-func TestParseTime(t *testing.T) {
- fixtures := []struct {
- input string
- output string
- }{
- {"2018-04-07T05:08:53.200Z", "2018-04-07 05:08:53.2 +0000 UTC"},
- {"1523077733", "2018-04-07 05:08:53 +0000 UTC"},
- {"1523077733.2", "2018-04-07 05:08:53.200000047 +0000 UTC"},
- }
-
- for _, f := range fixtures {
- out, err := parseTime(f.input)
- if err != nil {
- t.Error(err)
- }
-
- outStr := out.UTC().String()
- if outStr != f.output {
- t.Errorf("Expected %s, got %s for input %s", f.output, outStr, f.input)
- }
- }
-}
-
-func newTestTricksterHandler(t *testing.T) (tr *TricksterHandler, close func(t *testing.T)) {
- conf := NewConfig()
-
- conf.Origins["default"] = PrometheusOriginConfig{
- OriginURL: nonexistantOrigin,
- APIPath: prometheusAPIv1Path,
- IgnoreNoCacheHeader: true,
- MaxValueAgeSecs: 86400,
- }
- tr = &TricksterHandler{
- ResponseChannels: make(map[string]chan *ClientRequestContext),
- Config: conf,
- Logger: log.NewNopLogger(),
- Metrics: NewApplicationMetrics(),
- }
-
- tr.Cacher = getCache(tr)
- if err := tr.Cacher.Connect(); err != nil {
- t.Fatal("Unable to connect to cache:", err)
- }
-
- return tr, func(t *testing.T) {
- tr.Metrics.Unregister()
- if err := tr.Cacher.Close(); err != nil {
- t.Fatal("Error closing cacher:", err)
- }
- }
-}
-
-func (t *TricksterHandler) setTestOrigin(originURL string) {
- conf := NewConfig()
- conf.Origins["default"] = PrometheusOriginConfig{
- OriginURL: originURL,
- APIPath: prometheusAPIv1Path,
- IgnoreNoCacheHeader: true,
- MaxValueAgeSecs: 86400,
- }
- t.Config = conf
-}
-
-func TestUnreachableOriginReturnsStatusBadGateway(t *testing.T) {
- tests := []struct {
- handler func(*TricksterHandler, http.ResponseWriter, *http.Request)
- path string
- }{
- {
- handler: (*TricksterHandler).promHealthCheckHandler,
- },
- {
- handler: (*TricksterHandler).promFullProxyHandler,
- },
- {
- handler: (*TricksterHandler).promQueryHandler,
- },
- {
- handler: (*TricksterHandler).promQueryRangeHandler,
- path: prometheusAPIv1Path + "query_range?start=100000000&end=200000000&step=15&query=up",
- },
- }
-
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
-
- for _, test := range tests {
- rr := httptest.NewRecorder()
- test.handler(tr, rr, httptest.NewRequest("GET", "http://trickster"+test.path, nil))
- if rr.Result().StatusCode != http.StatusBadGateway {
- t.Errorf("unexpected status code; want %d, got %d", http.StatusBadGateway, rr.Result().StatusCode)
- }
- }
-}
-
-func TestMissingRangeQueryParametersResultInStatusBadRequest(t *testing.T) {
- paramsTests := []string{
- "start=0&end=100000000&query=up",
- "end=100000000&step=15&query=up",
- "start=0&step=15&query=up",
- }
-
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
-
- for _, params := range paramsTests {
- rr := httptest.NewRecorder()
- tr.promQueryRangeHandler(rr, httptest.NewRequest("GET", "http://trickster"+prometheusAPIv1Path+"query_range?"+params, nil))
- if rr.Result().StatusCode != http.StatusBadRequest {
- t.Errorf("unexpected status code for params %q; want %d, got %d", params, http.StatusBadRequest, rr.Result().StatusCode)
- }
- }
-}
-
-func newTestServer(body string) *httptest.Server {
- handler := func(w http.ResponseWriter, r *http.Request) {
- fmt.Fprint(w, body)
- }
- s := httptest.NewServer(http.HandlerFunc(handler))
- return s
-}
-
-func TestTricksterHandler_pingHandler(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- es := newTestServer("{}")
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- w := httptest.NewRecorder()
- r := httptest.NewRequest("GET", es.URL, nil)
- tr.pingHandler(w, r)
-
- resp := w.Result()
-
- // it should return 200 OK
- if resp.StatusCode != 200 {
- t.Errorf("wanted 200 got %d.", resp.StatusCode)
- }
-
- bodyBytes, err := ioutil.ReadAll(resp.Body)
- if err != nil {
- t.Error(err)
- }
-
- if string(bodyBytes) != "pong" {
- t.Errorf("wanted 'pong' got %s.", bodyBytes)
- }
-
-}
-
-func TestTricksterHandler_getOrigin(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
-
- // it should get test origin
- r := httptest.NewRequest("GET", nonexistantOrigin, nil)
- o := tr.getOrigin(r)
- if o.OriginURL != nonexistantOrigin {
- t.Errorf("wanted \"%s\" got \"%s\".", nonexistantOrigin, o.OriginURL)
- }
-}
-
-func TestTricksterHandler_promHealthCheckHandler(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- es := newTestServer("{}")
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should proxy request
- w := httptest.NewRecorder()
- r := httptest.NewRequest("GET", es.URL, nil)
- tr.promHealthCheckHandler(w, r)
-
- if w.Result().StatusCode != 200 {
- t.Errorf("wanted 200 got %d.", w.Result().StatusCode)
- }
-}
-
-func TestTricksterHandler_promFullProxyHandler(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- es := newTestServer("{}")
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should proxy request
- w := httptest.NewRecorder()
- r := httptest.NewRequest("GET", es.URL, nil)
- tr.promFullProxyHandler(w, r)
-
- if w.Result().StatusCode != 200 {
- t.Errorf("wanted 200 got %d.", w.Result().StatusCode)
- }
-}
-
-func TestTricksterHandler_promQueryHandler(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- es := newTestServer("{}")
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should proxy request
- w := httptest.NewRecorder()
- r := httptest.NewRequest("GET", es.URL, nil)
- tr.promQueryHandler(w, r)
-
- if w.Result().StatusCode != 200 {
- t.Errorf("wanted 200 got %d.", w.Result().StatusCode)
- }
-}
-
-func TestTricksterHandler_promQueryRangeHandler_cacheMiss(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- es := newTestServer(exampleRangeResponse)
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should queue the proxy request
- w := httptest.NewRecorder()
- r := httptest.NewRequest("GET", es.URL+exampleRangeQuery, nil)
- tr.promQueryRangeHandler(w, r)
-
- if w.Result().StatusCode != 200 {
- t.Errorf("wanted 200 got %d.", w.Result().StatusCode)
- }
-}
-
-func TestTricksterHandler_promQueryRangeHandler_cacheHit(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- es := newTestServer(exampleRangeResponse)
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // setup cache
- r := httptest.NewRequest("GET", es.URL+exampleRangeQuery, nil)
- tr.fetchPromQuery(es.URL+prometheusAPIv1Path+exampleRangeQuery_step, r.URL.Query(), r)
-
- // it should respond from cache
- w := httptest.NewRecorder()
- r = httptest.NewRequest("GET", es.URL+exampleRangeQuery, nil)
- tr.promQueryRangeHandler(w, r)
-
- resp := w.Result()
- defer resp.Body.Close()
-
- if resp.StatusCode != 200 {
- t.Errorf("wanted 200. got %d.", resp.StatusCode)
- }
-
- bodyBytes, err := ioutil.ReadAll(resp.Body)
- if err != nil {
- t.Error(err)
- }
-
- fmt.Println(string(bodyBytes))
-
- pm := PrometheusMatrixEnvelope{}
- err = json.Unmarshal(bodyBytes, &pm)
- if err != nil {
- t.Error(err)
- }
-
- if pm.getValueCount() != 6 {
- t.Errorf("wanted 6 got %d.", pm.getValueCount())
- }
-}
-
-func TestTricksterHandler_getURL(t *testing.T) {
- tr, closeFn := newTestTricksterHandler(t)
- defer closeFn(t)
- body := "{}"
- es := newTestServer(body)
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should get from the echo server
- b, _, _, err := tr.getURL(tr.Config.Origins["default"], "GET", es.URL, url.Values{}, nil)
- if err != nil {
- t.Error(err)
- }
- if bytes.Compare(b, []byte(body)) != 0 {
- t.Errorf("wanted \"%s\" got \"%s\"", body, b)
- }
-}
-
-func TestTricksterHandler_getVectorFromPrometheus(t *testing.T) {
- tr, closeTr := newTestTricksterHandler(t)
- defer closeTr(t)
- es := newTestServer(exampleResponse)
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should get an empty vector envelope
- r := httptest.NewRequest("GET", es.URL+exampleQuery, nil)
- pe, _, _, err := tr.getVectorFromPrometheus(es.URL, r.URL.Query(), r)
- if err != nil {
- t.Error(err)
- }
- if pe.Status != "success" {
- t.Errorf("wanted \"success\" got \"%s\".", pe.Status)
- }
-}
-
-func TestTricksterHandler_getMatrixFromPrometheus(t *testing.T) {
- tr, closeTr := newTestTricksterHandler(t)
- defer closeTr(t)
- es := newTestServer(exampleRangeResponse)
- defer es.Close()
- tr.setTestOrigin(es.URL)
-
- // it should get an empty matrix envelope
- r := httptest.NewRequest("GET", es.URL+exampleRangeQuery, nil)
- pe, _, _, _, err := tr.getMatrixFromPrometheus(es.URL, r.URL.Query(), r)
- if err != nil {
- t.Error(err)
- }
- if pe.Status != "success" {
- t.Errorf("wanted \"success\" got \"%s\".", pe.Status)
- }
-}
-
-func TestTricksterHandler_respondToCacheHit(t *testing.T) {
- tr, closeTr := newTestTricksterHandler(t)
- defer closeTr(t)
-
- w := httptest.NewRecorder()
- r := httptest.NewRequest("GET", nonexistantOrigin+exampleRangeQuery, nil)
- ctx, err := tr.buildRequestContext(w, r)
- if err != nil {
- t.Error(err)
- }
-
- // it should update the response in ctx.Writer without failing
- ctx.WaitGroup.Add(1)
- tr.respondToCacheHit(ctx)
-}
-
-func TestPrometheusMatrixEnvelope_getValueCount(t *testing.T) {
- pm := PrometheusMatrixEnvelope{}
- err := json.Unmarshal([]byte(exampleRangeResponse), &pm)
- if err != nil {
- t.Error(err)
- }
-
- // it should count the values in the matrix
- if 6 != pm.getValueCount() {
- t.Errorf("wanted 6 got %d.", pm.getValueCount())
- }
-}
-
-func TestTricksterHandler_mergeVector(t *testing.T) {
- tr, closeTr := newTestTricksterHandler(t)
- defer closeTr(t)
-
- pm := PrometheusMatrixEnvelope{}
- err := json.Unmarshal([]byte(exampleRangeResponse), &pm)
- if err != nil {
- t.Error(err)
- }
-
- pv := PrometheusVectorEnvelope{}
- err = json.Unmarshal([]byte(exampleResponse), &pv)
- if err != nil {
- t.Error(err)
- }
-
- // it should merge the values from the vector into the matrix
- pe := tr.mergeVector(pm, pv)
-
- if 8 != pe.getValueCount() {
- t.Errorf("wanted 8 got %d.", pe.getValueCount())
- }
-}
-
-func TestTricksterHandler_mergeMatrix(t *testing.T) {
- tests := []struct {
- a, b, merged PrometheusMatrixEnvelope
- }{
- // Series that adhere to rule
- {
- a: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{10, 1.5},
- },
- },
- },
- },
- },
- b: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{1, 1.5},
- model.SamplePair{5, 1.5},
- },
- },
- },
- },
- },
- merged: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{1, 1.5},
- model.SamplePair{5, 1.5},
- model.SamplePair{10, 1.5},
- },
- },
- },
- },
- },
- },
- // Empty second series
- {
- a: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{10, 1.5},
- },
- },
- },
- },
- },
- b: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{},
- },
- },
- },
- },
- merged: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{10, 1.5},
- },
- },
- },
- },
- },
- },
- // Series that have too many points in the second series
- {
- a: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{10, 1.5},
- },
- },
- },
- },
- },
- b: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{1, 1.5},
- model.SamplePair{5, 1.5},
- model.SamplePair{10, 1.5},
- model.SamplePair{15, 1.5},
- },
- },
- },
- },
- },
- merged: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{1, 1.5},
- model.SamplePair{5, 1.5},
- model.SamplePair{10, 1.5},
- },
- },
- },
- },
- },
- },
- // Series that don't adhere to rules
- {
- a: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{1, 1.5},
- },
- },
- },
- },
- },
- b: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{2, 1.5},
- },
- },
- },
- },
- },
- merged: PrometheusMatrixEnvelope{
- Status: rvSuccess,
- Data: PrometheusMatrixData{
- ResultType: "matrix",
- Result: model.Matrix{
- &model.SampleStream{
- Metric: model.Metric{"__name__": "a"},
- Values: []model.SamplePair{
- model.SamplePair{1, 1.5},
- },
- },
- },
- },
- },
- },
- }
-
- tr, closeTr := newTestTricksterHandler(t)
- defer closeTr(t)
-
- for i, test := range tests {
- t.Run(strconv.Itoa(i), func(t *testing.T) {
- merged := tr.mergeMatrix(test.a, test.b)
- if !reflect.DeepEqual(merged, test.merged) {
- t.Fatalf("Mismatch\nactual=%v\nexpected=%v", merged, test.merged)
- }
- })
- }
-}
-
-func TestAlignStepBoundaries(t *testing.T) {
- tests := []struct {
- start, end, stepMS, now int64
- rangeStart, rangeEnd int64
- err bool
- }{
- // Basic test
- {
- 1, 100, 1, 1,
- 1, 100,
- false,
- },
- // Ensure that it aligns to the step interval
- {
- 1, 100, 10, 1,
- 0, 100,
- false,
- },
- // query with start after end, ensure that it returns an error
- {
- 100, 1, 10, 1,
- 0, 0,
- true,
- },
- // query with start and end after now
- {
- 2000, 3000, 10, 1,
- 0, 0,
- true,
- },
- // query with start and end after now, with start after end
- {
- 3000, 2000, 10, 1,
- 0, 0,
- true,
- },
- // query with no step
- {
- 1, 100, 0, 1,
- 0, 0,
- true,
- },
- // query with negative step
- {
- 1, 100, -10, 1,
- 0, 0,
- true,
- },
- }
-
- for i, test := range tests {
- t.Run(strconv.Itoa(i), func(t *testing.T) {
- s, e, err := alignStepBoundaries(test.start, test.end, test.stepMS, test.now)
- if hasErr := err != nil; hasErr != test.err {
- t.Fatalf("Mismatch in error: expected=%v actual=%v", test.err, hasErr)
- }
- if s != test.rangeStart {
- t.Fatalf("Mismatch in rangeStart: expected=%d actual=%d", test.rangeStart, s)
- }
- if e != test.rangeEnd {
- t.Fatalf("Mismatch in rangeStart: expected=%d actual=%d", test.rangeEnd, e)
- }
- })
- }
-}
diff --git a/internal/cache/badger/badger.go b/internal/cache/badger/badger.go
new file mode 100644
index 000000000..ff7efb68d
--- /dev/null
+++ b/internal/cache/badger/badger.go
@@ -0,0 +1,151 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package badger
+
+import (
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/dgraph-io/badger"
+)
+
+// Cache describes a Badger Cache
+type Cache struct {
+ Name string
+ Config *config.CachingConfig
+ dbh *badger.DB
+}
+
+// Configuration returns the Configuration for the Cache object
+func (c *Cache) Configuration() *config.CachingConfig {
+ return c.Config
+}
+
+// Connect opens the configured Badger key-value store
+func (c *Cache) Connect() error {
+ log.Info("badger cache setup", log.Pairs{"cacheDir": c.Config.Badger.Directory})
+
+ opts := badger.DefaultOptions(c.Config.Badger.Directory)
+ opts.ValueDir = c.Config.Badger.ValueDirectory
+
+ var err error
+ c.dbh, err = badger.Open(opts)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// Store places the the data into the Badger Cache using the provided Key and TTL
+func (c *Cache) Store(cacheKey string, data []byte, ttl time.Duration) error {
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "set", "none", float64(len(data)))
+ log.Debug("badger cache store", log.Pairs{"key": cacheKey, "ttl": ttl})
+ return c.dbh.Update(func(txn *badger.Txn) error {
+ return txn.SetEntry(&badger.Entry{Key: []byte(cacheKey), Value: data, ExpiresAt: uint64(time.Now().Add(ttl).Unix())})
+ })
+}
+
+// Retrieve gets data from the Badger Cache using the provided Key
+// because Badger manages Object Expiration internally, allowExpired is not used.
+func (c *Cache) Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error) {
+ var data []byte
+ err := c.dbh.View(func(txn *badger.Txn) error {
+ item, err := txn.Get([]byte(cacheKey))
+ if err != nil {
+ return err
+ }
+ data, err = item.ValueCopy(nil)
+ return err
+
+ })
+
+ if err == nil {
+ log.Debug("badger cache retrieve", log.Pairs{"key": cacheKey})
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "get", "hit", float64(len(data)))
+ return data, status.LookupStatusHit, nil
+ }
+
+ if err == badger.ErrKeyNotFound {
+ log.Debug("badger cache miss", log.Pairs{"key": cacheKey})
+ cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ return nil, status.LookupStatusKeyMiss, err
+ }
+
+ log.Debug("badger cache retrieve failed", log.Pairs{"key": cacheKey, "reason": err.Error()})
+ cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ return data, status.LookupStatusError, err
+}
+
+// Remove removes an object in cache, if present
+func (c *Cache) Remove(cacheKey string) {
+ log.Debug("badger cache remove", log.Pairs{"key": cacheKey})
+ c.dbh.Update(func(txn *badger.Txn) error {
+ return txn.Delete([]byte(cacheKey))
+ })
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, 0)
+}
+
+// BulkRemove removes a list of objects from the cache. noLock is not used for Badger
+func (c *Cache) BulkRemove(cacheKeys []string, noLock bool) {
+ log.Debug("badger cache bulk remove", log.Pairs{})
+
+ c.dbh.Update(func(txn *badger.Txn) error {
+ for _, key := range cacheKeys {
+ if err := txn.Delete([]byte(key)); err != nil {
+ return err
+ }
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, 0)
+ }
+ return nil
+ })
+}
+
+// Close closes the Badger Cache
+func (c *Cache) Close() error {
+ return c.dbh.Close()
+}
+
+// SetTTL updates the TTL for the provided cache object
+func (c *Cache) SetTTL(cacheKey string, ttl time.Duration) {
+ var data []byte
+ err := c.dbh.Update(func(txn *badger.Txn) error {
+ item, err := txn.Get([]byte(cacheKey))
+ if err != nil {
+ return nil
+ }
+ data, _ = item.ValueCopy(nil)
+ return txn.SetEntry(&badger.Entry{Key: []byte(cacheKey), Value: data, ExpiresAt: uint64(time.Now().Add(ttl).Unix())})
+ })
+ log.Debug("badger cache update-ttl", log.Pairs{"key": cacheKey, "ttl": ttl, "success": err == nil})
+ if err == nil {
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "update-ttl", "none", 0)
+ }
+}
+
+func (c *Cache) getExpires(cacheKey string) (int, error) {
+ var expires int
+ err := c.dbh.View(func(txn *badger.Txn) error {
+ item, err := txn.Get([]byte(cacheKey))
+ if err != nil {
+ return err
+ }
+ expires = int(item.ExpiresAt())
+ return nil
+ })
+ return expires, err
+}
diff --git a/internal/cache/badger/badger_test.go b/internal/cache/badger/badger_test.go
new file mode 100644
index 000000000..a3b574644
--- /dev/null
+++ b/internal/cache/badger/badger_test.go
@@ -0,0 +1,275 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package badger
+
+import (
+ "io/ioutil"
+ "os"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+const cacheType = "badger"
+const cacheKey = "cacheKey"
+
+func newCacheConfig(t *testing.T) config.CachingConfig {
+ dir, err := ioutil.TempDir("/tmp", cacheType)
+ if err != nil {
+ t.Fatalf("could not create temp directory (%s): %s", dir, err)
+ }
+ return config.CachingConfig{CacheType: cacheType, Badger: config.BadgerCacheConfig{Directory: dir, ValueDirectory: dir}}
+}
+
+func TestConfiguration(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ cfg := bc.Configuration()
+ if cfg.CacheType != cacheType {
+ t.Fatalf("expected %s got %s", cacheType, cfg.CacheType)
+ }
+}
+
+func TestBadgerCache_Connect(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ // it should connect
+ if err := bc.Connect(); err != nil {
+ t.Error(err)
+ }
+ bc.Close()
+}
+
+func TestBadgerCache_ConnectFailed(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ cacheConfig.Badger.Directory = "/root/trickster-test-noaccess"
+ os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ // it should connect
+ err := bc.Connect()
+ if err == nil {
+ t.Errorf("expected file access error for %s", cacheConfig.Badger.Directory)
+ bc.Close()
+ }
+}
+
+func TestBadgerCache_Store(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ if err := bc.Connect(); err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ err := bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestBadgerCache_Remove(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ if err := bc.Connect(); err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ err := bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := bc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+
+ bc.Remove(cacheKey)
+
+ // it should be a cache miss
+ _, ls, err = bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func TestBadgerCache_BulkRemove(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ if err := bc.Connect(); err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ err := bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := bc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ bc.BulkRemove([]string{""}, true)
+ bc.BulkRemove([]string{cacheKey}, true)
+
+ // it should be a cache miss
+ _, ls, err = bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+}
+
+func TestBadgerCache_Retrieve(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Badger.Directory)
+ bc := Cache{Config: &cacheConfig}
+
+ if err := bc.Connect(); err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should be a cache miss
+ _, ls, err := bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ err = bc.Store(cacheKey, []byte("data"), time.Duration(5)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := bc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+
+ exp1, err := bc.getExpires(cacheKey)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // the TTL is currently 1 second. update it to 1 hour then wait more than
+ // 1 second, to ensure it remained in cache with the correct value
+ bc.SetTTL(cacheKey, time.Duration(3600)*time.Second)
+
+ exp2, err := bc.getExpires(cacheKey)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // should be around 3595
+ diff := exp2 - exp1
+ const expected = 3590
+
+ if diff < 3590 {
+ t.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, exp2, exp1)
+ }
+
+ // try a non-existent cacheKey
+ ck2 := cacheKey + "xxxx"
+ bc.SetTTL(ck2, time.Duration(3600)*time.Second)
+
+ // it should be a cache miss
+ _, ls, err = bc.Retrieve(ck2, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", ck2)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ // it should also not have an expires
+ _, err = bc.getExpires(ck2)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", ck2)
+ }
+}
+
+func TestBadgerCache_Close(t *testing.T) {
+ dir, err := ioutil.TempDir("/tmp", cacheType)
+ if err != nil {
+ t.Fatalf("could not create temp directory (%s): %s", dir, err)
+ }
+ defer os.RemoveAll(dir)
+
+ cacheConfig := config.CachingConfig{CacheType: cacheType, Badger: config.BadgerCacheConfig{Directory: dir, ValueDirectory: dir}}
+ bc := Cache{Config: &cacheConfig}
+
+ if err := bc.Connect(); err != nil {
+ t.Error(err)
+ }
+
+ // it should close
+ if err := bc.Close(); err != nil {
+ t.Error(err)
+ }
+}
diff --git a/internal/cache/bbolt/bbolt.go b/internal/cache/bbolt/bbolt.go
new file mode 100644
index 000000000..061a452a0
--- /dev/null
+++ b/internal/cache/bbolt/bbolt.go
@@ -0,0 +1,207 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package bbolt
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/coreos/bbolt"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/index"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/pkg/locks"
+)
+
+var lockPrefix string
+
+// Cache describes a BBolt Cache
+type Cache struct {
+ Name string
+ Config *config.CachingConfig
+ dbh *bbolt.DB
+ Index *index.Index
+}
+
+// Configuration returns the Configuration for the Cache object
+func (c *Cache) Configuration() *config.CachingConfig {
+ return c.Config
+}
+
+// Connect instantiates the Cache mutex map and starts the Expired Entry Reaper goroutine
+func (c *Cache) Connect() error {
+ log.Info("bbolt cache setup", log.Pairs{"name": c.Name, "cacheFile": c.Config.BBolt.Filename})
+
+ lockPrefix = c.Name + ".bbolt."
+
+ var err error
+ c.dbh, err = bbolt.Open(c.Config.BBolt.Filename, 0644, &bbolt.Options{Timeout: 1 * time.Second})
+ if err != nil {
+ return err
+ }
+
+ err = c.dbh.Update(func(tx *bbolt.Tx) error {
+ _, err2 := tx.CreateBucketIfNotExists([]byte(c.Config.BBolt.Bucket))
+ if err2 != nil {
+ return fmt.Errorf("create bucket: %s", err2)
+ }
+ return nil
+ })
+ if err != nil {
+ return err
+ }
+
+ // Load Index here and pass bytes as param2
+ indexData, _, _ := c.retrieve(index.IndexKey, false, false)
+ c.Index = index.NewIndex(c.Name, c.Config.CacheType, indexData, c.Config.Index, c.BulkRemove, c.storeNoIndex)
+ return nil
+}
+
+// Store places an object in the cache using the specified key and ttl
+func (c *Cache) Store(cacheKey string, data []byte, ttl time.Duration) error {
+ return c.store(cacheKey, data, ttl, true)
+}
+
+func (c *Cache) storeNoIndex(cacheKey string, data []byte) {
+ err := c.store(cacheKey, data, 31536000*time.Second, false)
+ if err != nil {
+ log.Error("cache failed to write non-indexed object", log.Pairs{"cacheName": c.Name, "cacheType": "bbolt", "cacheKey": cacheKey, "objectSize": len(data)})
+ }
+}
+
+func (c *Cache) store(cacheKey string, data []byte, ttl time.Duration, updateIndex bool) error {
+
+ locks.Acquire(lockPrefix + cacheKey)
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "set", "none", float64(len(data)))
+
+ o := &index.Object{Key: cacheKey, Value: data, Expiration: time.Now().Add(ttl)}
+ err := writeToBBolt(c.dbh, c.Config.BBolt.Bucket, cacheKey, o.ToBytes())
+ if err != nil {
+ locks.Release(lockPrefix + cacheKey)
+ return err
+ }
+ log.Debug("bbolt cache store", log.Pairs{"key": cacheKey, "ttl": ttl, "indexed": updateIndex})
+ if updateIndex {
+ c.Index.UpdateObject(o)
+ }
+ locks.Release(lockPrefix + cacheKey)
+ return nil
+}
+
+func writeToBBolt(dbh *bbolt.DB, bucketName, cacheKey string, data []byte) error {
+ err := dbh.Update(func(tx *bbolt.Tx) error {
+ b := tx.Bucket([]byte(bucketName))
+ err2 := b.Put([]byte(cacheKey), data)
+ locks.Release(lockPrefix + cacheKey)
+ return err2
+ })
+ return err
+}
+
+// Retrieve looks for an object in cache and returns it (or an error if not found)
+func (c *Cache) Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error) {
+ return c.retrieve(cacheKey, allowExpired, true)
+}
+
+func (c *Cache) retrieve(cacheKey string, allowExpired bool, atime bool) ([]byte, status.LookupStatus, error) {
+
+ locks.Acquire(lockPrefix + cacheKey)
+
+ var data []byte
+ err := c.dbh.View(func(tx *bbolt.Tx) error {
+ b := tx.Bucket([]byte(c.Config.BBolt.Bucket))
+ data = b.Get([]byte(cacheKey))
+ if data == nil {
+ log.Debug("bbolt cache miss", log.Pairs{"key": cacheKey})
+ _, cme := cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ locks.Release(lockPrefix + cacheKey)
+ return cme
+ }
+ locks.Release(lockPrefix + cacheKey)
+ return nil
+ })
+ if err != nil {
+ locks.Release(lockPrefix + cacheKey)
+ return nil, status.LookupStatusKeyMiss, err
+ }
+
+ o, err := index.ObjectFromBytes(data)
+ if err != nil {
+ locks.Release(lockPrefix + cacheKey)
+ _, err = cache.CacheError(cacheKey, c.Name, c.Config.CacheType, "value for key [%s] could not be deserialized from cache")
+ return nil, status.LookupStatusError, err
+ }
+ o.Expiration = c.Index.GetExpiration(cacheKey)
+
+ if allowExpired || o.Expiration.IsZero() || o.Expiration.After(time.Now()) {
+ log.Debug("bbolt cache retrieve", log.Pairs{"cacheKey": cacheKey})
+ if atime {
+ c.Index.UpdateObjectAccessTime(cacheKey)
+ }
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "get", "hit", float64(len(data)))
+ locks.Release(lockPrefix + cacheKey)
+ return o.Value, status.LookupStatusHit, nil
+ }
+ // Cache Object has been expired but not reaped, go ahead and delete it
+ c.remove(cacheKey, false)
+ b, err := cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ locks.Release(lockPrefix + cacheKey)
+
+ return b, status.LookupStatusKeyMiss, err
+}
+
+// SetTTL updates the TTL for the provided cache object
+func (c *Cache) SetTTL(cacheKey string, ttl time.Duration) {
+ locks.Acquire(lockPrefix + cacheKey)
+ c.Index.UpdateObjectTTL(cacheKey, ttl)
+ locks.Release(lockPrefix + cacheKey)
+}
+
+// Remove removes an object in cache, if present
+func (c *Cache) Remove(cacheKey string) {
+ locks.Acquire(lockPrefix + cacheKey)
+ c.remove(cacheKey, false)
+ locks.Release(lockPrefix + cacheKey)
+}
+
+func (c *Cache) remove(cacheKey string, noLock bool) error {
+
+ err := c.dbh.Update(func(tx *bbolt.Tx) error {
+ b := tx.Bucket([]byte(c.Config.BBolt.Bucket))
+ return b.Delete([]byte(cacheKey))
+ })
+ if err != nil {
+ log.Error("bbolt cache key delete failure", log.Pairs{"cacheKey": cacheKey, "reason": err.Error()})
+ return err
+ }
+ c.Index.RemoveObject(cacheKey, noLock)
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, 0)
+ log.Debug("bbolt cache key delete", log.Pairs{"key": cacheKey})
+ return nil
+}
+
+// BulkRemove removes a list of objects from the cache
+func (c *Cache) BulkRemove(cacheKeys []string, noLock bool) {
+ for _, cacheKey := range cacheKeys {
+ c.remove(cacheKey, noLock)
+ }
+}
+
+// Close closes the Cache
+func (c *Cache) Close() error {
+ return c.dbh.Close()
+}
diff --git a/internal/cache/bbolt/bbolt_test.go b/internal/cache/bbolt/bbolt_test.go
new file mode 100644
index 000000000..f6037d416
--- /dev/null
+++ b/internal/cache/bbolt/bbolt_test.go
@@ -0,0 +1,581 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package bbolt
+
+import (
+ "os"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/util/log"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+const cacheType = "bbolt"
+const cacheKey = "cacheKey"
+
+func newCacheConfig() config.CachingConfig {
+ const testDbPath = "/tmp/test.db"
+ os.Remove(testDbPath)
+ return config.CachingConfig{CacheType: cacheType, BBolt: config.BBoltCacheConfig{Filename: testDbPath, Bucket: "trickster_test"}, Index: config.CacheIndexConfig{ReapInterval: time.Second}}
+}
+
+func storeBenchmark(b *testing.B) Cache {
+ log.Logger = log.ConsoleLogger("none")
+ testDbPath := "/tmp/test.db"
+ os.Remove(testDbPath)
+ cacheConfig := config.CachingConfig{CacheType: cacheType, BBolt: config.BBoltCacheConfig{Filename: testDbPath, Bucket: "trickster_test"}, Index: config.CacheIndexConfig{ReapInterval: time.Second}}
+ bc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := bc.Connect()
+ if err != nil {
+ b.Error(err)
+ }
+
+ // it should store a value
+ for n := 0; n < b.N; n++ {
+ err = bc.Store(cacheKey+strconv.Itoa(n), []byte("data"+strconv.Itoa(n)), time.Duration(60)*time.Second)
+ if err != nil {
+ b.Error(err)
+ }
+ }
+ return bc
+}
+
+func TestConfiguration(t *testing.T) {
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+ cfg := bc.Configuration()
+ if cfg.CacheType != cacheType {
+ t.Errorf("expected %s got %s", cacheType, cfg.CacheType)
+ }
+}
+
+func TestBboltCache_Connect(t *testing.T) {
+ cacheConfig := newCacheConfig()
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+ bc := Cache{Config: &cacheConfig}
+ // it should connect
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ bc.Close()
+}
+
+func TestBboltCache_ConnectFailed(t *testing.T) {
+ const expected = `open /root/noaccess.bbolt:`
+ cacheConfig := newCacheConfig()
+ cacheConfig.BBolt.Filename = "/root/noaccess.bbolt"
+ bc := Cache{Config: &cacheConfig}
+ // it should connect
+ err := bc.Connect()
+ if err == nil {
+ t.Errorf("expected error for %s", expected)
+ bc.Close()
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+ }
+ if !strings.HasPrefix(err.Error(), expected) {
+ t.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+}
+
+func TestBboltCache_ConnectBadBucketName(t *testing.T) {
+ const expected = `create bucket: bucket name required`
+ cacheConfig := newCacheConfig()
+ cacheConfig.BBolt.Bucket = ""
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+ bc := Cache{Config: &cacheConfig}
+ // it should connect
+ err := bc.Connect()
+ if err == nil {
+ t.Errorf("expected error for %s", expected)
+ bc.Close()
+ }
+ if err.Error() != expected {
+ t.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+}
+
+func TestBboltCache_Store(t *testing.T) {
+
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ err = bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkCache_Store(b *testing.B) {
+ bc := storeBenchmark(b)
+ defer bc.Close()
+}
+
+func TestBboltCache_SetTTL(t *testing.T) {
+
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ exp1 := bc.Index.GetExpiration(cacheKey)
+ if !exp1.IsZero() {
+ t.Errorf("expected Zero time, got %v", exp1)
+ }
+
+ // it should store a value
+ err = bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ exp1 = bc.Index.GetExpiration(cacheKey)
+ if exp1.IsZero() {
+ t.Errorf("expected time %d, got zero", int(time.Now().Unix())+60)
+ }
+
+ e1 := int(exp1.Unix())
+
+ bc.SetTTL(cacheKey, time.Duration(3600)*time.Second)
+
+ exp2 := bc.Index.GetExpiration(cacheKey)
+ if exp2.IsZero() {
+ t.Errorf("expected time %d, got zero", int(time.Now().Unix())+3600)
+ }
+ e2 := int(exp2.Unix())
+
+ // should be around 3595
+ diff := e2 - e1
+ const expected = 3500
+
+ if diff < expected {
+ t.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, e2, e1)
+ }
+}
+
+func BenchmarkCache_SetTTL(b *testing.B) {
+ bc := storeBenchmark(b)
+ defer bc.Close()
+ for n := 0; n < b.N; n++ {
+ exp1 := bc.Index.GetExpiration(cacheKey + strconv.Itoa(n))
+ if exp1.IsZero() {
+ b.Errorf("expected time %d, got zero", int(time.Now().Unix())+60)
+ }
+
+ e1 := int(exp1.Unix())
+
+ bc.SetTTL(cacheKey+strconv.Itoa(n), time.Duration(3600)*time.Second)
+
+ exp2 := bc.Index.GetExpiration(cacheKey + strconv.Itoa(n))
+ if exp2.IsZero() {
+ b.Errorf("expected time %d, got zero", int(time.Now().Unix())+3600)
+ }
+ e2 := int(exp2.Unix())
+
+ // should be around 3595
+ diff := e2 - e1
+ const expected = 3500
+
+ if diff < expected {
+ b.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, e2, e1)
+ }
+ }
+}
+
+func TestBboltCache_StoreNoIndex(t *testing.T) {
+
+ const expected = `value for key [] not in cache`
+
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ bc.storeNoIndex(cacheKey, []byte("data"))
+
+ // it should retrieve a value
+ data, ls, err := bc.retrieve(cacheKey, false, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+
+ // test for error when bad key name
+ bc.storeNoIndex("", []byte("data"))
+
+ data, ls, err = bc.retrieve("", false, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected)
+ bc.Close()
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ if err.Error() != expected {
+ t.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+ if string(data) != "" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+}
+
+func BenchmarkCache_StoreNoIndex(b *testing.B) {
+ bc := storeBenchmark(b)
+ defer bc.Close()
+ for n := 0; n < b.N; n++ {
+ expected := `value for key [] not in cache`
+ // it should store a value
+ bc.storeNoIndex(cacheKey+strconv.Itoa(n), []byte("data"+strconv.Itoa(n)))
+
+ // it should retrieve a value
+ data, ls, err := bc.retrieve(cacheKey+strconv.Itoa(n), false, false)
+ if err != nil {
+ b.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+
+ // test for error when bad key name
+ bc.storeNoIndex("", []byte("data"+strconv.Itoa(n)))
+
+ data, ls, err = bc.retrieve("", false, false)
+ if err == nil {
+ b.Errorf("expected error for %s", expected)
+ bc.Close()
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ if err.Error() != expected {
+ b.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ }
+}
+
+func TestBboltCache_Remove(t *testing.T) {
+
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ err = bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := bc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+
+ bc.Remove(cacheKey)
+
+ // it should be a cache miss
+ _, ls, err = bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+}
+
+func BenchmarkCache_Remove(b *testing.B) {
+ bc := storeBenchmark(b)
+ defer bc.Close()
+ for n := 0; n < b.N; n++ {
+ var data []byte
+ data, ls, err := bc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\"", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ bc.Remove(cacheKey + strconv.Itoa(n))
+
+ // this should now return error
+ data, ls, err = bc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ expectederr := `value for key [` + cacheKey + strconv.Itoa(n) + `] not in cache`
+ if err == nil {
+ b.Errorf("expected error for %s", expectederr)
+ bc.Close()
+ }
+ if err.Error() != expectederr {
+ b.Errorf("expected error '%s' got '%s'", expectederr, err.Error())
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ }
+}
+
+func TestBboltCache_BulkRemove(t *testing.T) {
+
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ // it should store a value
+ err = bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := bc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ bc.BulkRemove([]string{cacheKey}, true)
+
+ // it should be a cache miss
+ _, ls, err = bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+}
+
+func BenchmarkCache_BulkRemove(b *testing.B) {
+ bc := storeBenchmark(b)
+ defer bc.Close()
+
+ var keyArray []string
+ for n := 0; n < b.N; n++ {
+ keyArray = append(keyArray, cacheKey+strconv.Itoa(n))
+ }
+
+ bc.BulkRemove(keyArray, true)
+
+ // it should be a cache miss
+ for n := 0; n < b.N; n++ {
+ _, ls, err := bc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestBboltCache_Retrieve(t *testing.T) {
+
+ const expected1 = `value for key [cacheKey] not in cache`
+ const expected2 = `value for key [cacheKey] could not be deserialized from cache`
+
+ cacheConfig := newCacheConfig()
+ bc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := bc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer bc.Close()
+
+ err = bc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := bc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // expire the object
+ bc.SetTTL(cacheKey, -1*time.Hour)
+
+ // this should now return error
+ data, ls, err = bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ bc.Close()
+ }
+ if err.Error() != expected1 {
+ t.Errorf("expected error '%s' got '%s'", expected1, err.Error())
+ }
+ if string(data) != "" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ // create a corrupted cache entry and expect an error
+ writeToBBolt(bc.dbh, cacheConfig.BBolt.Bucket, cacheKey, []byte("asdasdfasf"))
+
+ // it should fail to retrieve a value
+ data, ls, err = bc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected2)
+ bc.Close()
+ }
+ if err.Error() != expected2 {
+ t.Errorf("expected error '%s' got '%s'", expected2, err.Error())
+ }
+ if string(data) != "" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusError {
+ t.Errorf("expected %s got %s", status.LookupStatusError, ls)
+ }
+}
+
+func BenchmarkCache_Retrieve(b *testing.B) {
+ bc := storeBenchmark(b)
+ defer bc.Close()
+
+ for n := 0; n < b.N; n++ {
+ expected1 := `value for key [` + cacheKey + strconv.Itoa(n) + `] not in cache`
+ expected2 := `value for key [` + cacheKey + strconv.Itoa(n) + `] could not be deserialized from cache`
+
+ data, ls, err := bc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // expire the object
+ bc.SetTTL(cacheKey+strconv.Itoa(n), -1*time.Hour)
+
+ // this should now return error
+ data, ls, err = bc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected error for %s", expected1)
+ bc.Close()
+ }
+ if err.Error() != expected1 {
+ b.Errorf("expected error '%s' got '%s'", expected1, err.Error())
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ // create a corrupted cache entry and expect an error
+ writeToBBolt(bc.dbh, bc.Config.BBolt.Bucket, cacheKey+strconv.Itoa(n), []byte("asdasdfasf"+strconv.Itoa(n)))
+
+ // it should fail to retrieve a value
+ data, ls, err = bc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected error for %s", expected2)
+ bc.Close()
+ }
+ if err.Error() != expected2 {
+ b.Errorf("expected error '%s' got '%s'", expected2, err.Error())
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
diff --git a/internal/cache/cache.go b/internal/cache/cache.go
new file mode 100644
index 000000000..67b725f59
--- /dev/null
+++ b/internal/cache/cache.go
@@ -0,0 +1,97 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package cache
+
+import (
+ "errors"
+ "fmt"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+// ErrKNF represents the error "key not found in cache"
+var ErrKNF = errors.New("key not found in cache")
+
+// Cache is the interface for the supported caching fabrics
+// When making new cache types, Retrieve() must return an error on cache miss
+type Cache interface {
+ Connect() error
+ Store(cacheKey string, data []byte, ttl time.Duration) error
+ Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error)
+ SetTTL(cacheKey string, ttl time.Duration)
+ Remove(cacheKey string)
+ BulkRemove(cacheKeys []string, noLock bool)
+ Close() error
+ Configuration() *config.CachingConfig
+}
+
+// MemoryCache is the interface for an in-memory cache
+// This offers an additional method for storing references to bypass serialization
+type MemoryCache interface {
+ Connect() error
+ Store(cacheKey string, data []byte, ttl time.Duration) error
+ Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error)
+ SetTTL(cacheKey string, ttl time.Duration)
+ Remove(cacheKey string)
+ BulkRemove(cacheKeys []string, noLock bool)
+ Close() error
+ Configuration() *config.CachingConfig
+ StoreReference(cacheKey string, data ReferenceObject, ttl time.Duration) error
+ RetrieveReference(cacheKey string, allowExpired bool) (interface{}, status.LookupStatus, error)
+}
+
+// ReferenceObject defines an interface for a cache object possessing the ability to report
+// the approximate comprehensive byte size of its members, to assist with cache size management
+type ReferenceObject interface {
+ Size() int
+}
+
+// ObserveCacheMiss returns a standard Cache Miss response
+func ObserveCacheMiss(cacheKey, cacheName, cacheType string) ([]byte, error) {
+ ObserveCacheOperation(cacheName, cacheType, "get", "miss", 0)
+ return nil, fmt.Errorf("value for key [%s] not in cache", cacheKey)
+}
+
+// ObserveCacheDel records a cache deletion event
+func ObserveCacheDel(cache, cacheType string, count float64) {
+ ObserveCacheOperation(cache, cacheType, "del", "none", count)
+}
+
+// CacheError returns an empty cache object and the formatted error
+func CacheError(cacheKey, cacheName, cacheType string, msg string) ([]byte, error) {
+ ObserveCacheEvent(cacheName, cacheType, "error", msg)
+ return nil, fmt.Errorf(msg, cacheKey)
+}
+
+// ObserveCacheOperation increments counters as cache operations occur
+func ObserveCacheOperation(cache, cacheType, operation, status string, bytes float64) {
+ metrics.CacheObjectOperations.WithLabelValues(cache, cacheType, operation, status).Inc()
+ if bytes > 0 {
+ metrics.CacheByteOperations.WithLabelValues(cache, cacheType, operation, status).Add(float64(bytes))
+ }
+}
+
+// ObserveCacheEvent increments counters as cache events occur
+func ObserveCacheEvent(cache, cacheType, event, reason string) {
+ metrics.CacheEvents.WithLabelValues(cache, cacheType, event, reason).Inc()
+}
+
+// ObserveCacheSizeChange adjust counters and gauges as the cache size changes due to object operations
+func ObserveCacheSizeChange(cache, cacheType string, byteCount, objectCount int64) {
+ metrics.CacheObjects.WithLabelValues(cache, cacheType).Set(float64(objectCount))
+ metrics.CacheBytes.WithLabelValues(cache, cacheType).Set(float64(byteCount))
+}
diff --git a/internal/cache/cache_test.go b/internal/cache/cache_test.go
new file mode 100644
index 000000000..c7cb22183
--- /dev/null
+++ b/internal/cache/cache_test.go
@@ -0,0 +1,66 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package cache
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+var testCacheKey, testCacheName, testCacheType string
+
+func init() {
+
+ testCacheKey = "test-key"
+ testCacheName = "test-cache"
+ testCacheType = "test"
+
+ metrics.Init()
+
+}
+
+func TestObserveCacheMiss(t *testing.T) {
+ e1 := fmt.Errorf("value for key [%s] not in cache", testCacheKey)
+ _, err := ObserveCacheMiss(testCacheKey, testCacheName, testCacheType)
+ if err.Error() != e1.Error() {
+ t.Error(err)
+ }
+}
+
+// ObserveCacheDel records a cache deletion event
+func TestObserveCacheDel(t *testing.T) {
+ ObserveCacheDel(testCacheName, testCacheType, 0)
+}
+
+func TestCacheError(t *testing.T) {
+ _, err := CacheError(testCacheKey, testCacheName, testCacheType, "%s")
+ if err.Error() != testCacheKey {
+ t.Errorf("expected %s got %s", testCacheKey, err.Error())
+ }
+}
+
+func TestObserveCacheOperation(t *testing.T) {
+ ObserveCacheOperation(testCacheName, testCacheType, "set", "ok", 0)
+ ObserveCacheOperation(testCacheName, testCacheType, "set", "ok", 1)
+}
+
+func TestObserveCacheEvent(t *testing.T) {
+ ObserveCacheEvent(testCacheName, testCacheType, "test", "test")
+}
+
+func TestObserveCacheSizeChange(t *testing.T) {
+ ObserveCacheSizeChange(testCacheName, testCacheType, 0, 0)
+}
diff --git a/internal/cache/filesystem/filesystem.go b/internal/cache/filesystem/filesystem.go
new file mode 100644
index 000000000..93d45e88d
--- /dev/null
+++ b/internal/cache/filesystem/filesystem.go
@@ -0,0 +1,197 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package filesystem
+
+import (
+ "fmt"
+ "io/ioutil"
+ "os"
+ "strings"
+ "time"
+
+ "golang.org/x/sys/unix"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/index"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/pkg/locks"
+)
+
+var lockPrefix string
+
+// Cache describes a Filesystem Cache
+type Cache struct {
+ Name string
+ Config *config.CachingConfig
+ Index *index.Index
+}
+
+// Configuration returns the Configuration for the Cache object
+func (c *Cache) Configuration() *config.CachingConfig {
+ return c.Config
+}
+
+// Connect instantiates the Cache mutex map and starts the Expired Entry Reaper goroutine
+func (c *Cache) Connect() error {
+ log.Info("filesystem cache setup", log.Pairs{"name": c.Name, "cachePath": c.Config.Filesystem.CachePath})
+ if err := makeDirectory(c.Config.Filesystem.CachePath); err != nil {
+ return err
+ }
+ lockPrefix = c.Name + ".file."
+
+ // Load Index here and pass bytes as param2
+ indexData, _, _ := c.retrieve(index.IndexKey, false, false)
+ c.Index = index.NewIndex(c.Name, c.Config.CacheType, indexData, c.Config.Index, c.BulkRemove, c.storeNoIndex)
+ return nil
+}
+
+// Store places an object in the cache using the specified key and ttl
+func (c *Cache) Store(cacheKey string, data []byte, ttl time.Duration) error {
+ return c.store(cacheKey, data, ttl, true)
+}
+
+func (c *Cache) storeNoIndex(cacheKey string, data []byte) {
+ err := c.store(cacheKey, data, 31536000*time.Second, false)
+ if err != nil {
+ log.Error("cache failed to write non-indexed object", log.Pairs{"cacheName": c.Name, "cacheType": "filesystem", "cacheKey": cacheKey, "objectSize": len(data)})
+ }
+}
+
+func (c *Cache) store(cacheKey string, data []byte, ttl time.Duration, updateIndex bool) error {
+
+ if ttl < 1 {
+ return fmt.Errorf("invalid ttl: %d", int64(ttl.Seconds()))
+ }
+
+ if cacheKey == "" {
+ return fmt.Errorf("cacheKey required")
+ }
+
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "set", "none", float64(len(data)))
+
+ dataFile := c.getFileName(cacheKey)
+
+ locks.Acquire(lockPrefix + cacheKey)
+
+ o := &index.Object{Key: cacheKey, Value: data, Expiration: time.Now().Add(ttl)}
+ err := ioutil.WriteFile(dataFile, o.ToBytes(), os.FileMode(0777))
+ if err != nil {
+ locks.Release(lockPrefix + cacheKey)
+ return err
+ }
+ log.Debug("filesystem cache store", log.Pairs{"key": cacheKey, "dataFile": dataFile, "indexed": updateIndex})
+ if updateIndex {
+ c.Index.UpdateObject(o)
+ }
+ locks.Release(lockPrefix + cacheKey)
+ return nil
+
+}
+
+// Retrieve looks for an object in cache and returns it (or an error if not found)
+func (c *Cache) Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error) {
+ return c.retrieve(cacheKey, allowExpired, true)
+}
+
+func (c *Cache) retrieve(cacheKey string, allowExpired bool, atime bool) ([]byte, status.LookupStatus, error) {
+
+ dataFile := c.getFileName(cacheKey)
+
+ locks.Acquire(lockPrefix + cacheKey)
+
+ data, err := ioutil.ReadFile(dataFile)
+ if err != nil {
+ log.Debug("filesystem cache miss", log.Pairs{"key": cacheKey, "dataFile": dataFile})
+ b, err2 := cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ locks.Release(lockPrefix + cacheKey)
+ return b, status.LookupStatusKeyMiss, err2
+ }
+
+ o, err := index.ObjectFromBytes(data)
+ if err != nil {
+ locks.Release(lockPrefix + cacheKey)
+ _, err2 := cache.CacheError(cacheKey, c.Name, c.Config.CacheType, "value for key [%s] could not be deserialized from cache")
+ return nil, status.LookupStatusError, err2
+ }
+ o.Expiration = c.Index.GetExpiration(cacheKey)
+ if allowExpired || o.Expiration.IsZero() || o.Expiration.After(time.Now()) {
+ log.Debug("filesystem cache retrieve", log.Pairs{"key": cacheKey, "dataFile": dataFile})
+ if atime {
+ c.Index.UpdateObjectAccessTime(cacheKey)
+ }
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "get", "hit", float64(len(data)))
+ locks.Release(lockPrefix + cacheKey)
+ return o.Value, status.LookupStatusHit, nil
+ }
+ // Cache Object has been expired but not reaped, go ahead and delete it
+ c.remove(cacheKey, false)
+ b, err := cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ locks.Release(lockPrefix + cacheKey)
+ return b, status.LookupStatusKeyMiss, err
+
+}
+
+// SetTTL updates the TTL for the provided cache object
+func (c *Cache) SetTTL(cacheKey string, ttl time.Duration) {
+ c.Index.UpdateObjectTTL(cacheKey, ttl)
+}
+
+// Remove removes an object from the cache
+func (c *Cache) Remove(cacheKey string) {
+ locks.Acquire(lockPrefix + cacheKey)
+ c.remove(cacheKey, false)
+ locks.Release(lockPrefix + cacheKey)
+}
+
+func (c *Cache) remove(cacheKey string, noLock bool) {
+
+ if err := os.Remove(c.getFileName(cacheKey)); err == nil {
+ c.Index.RemoveObject(cacheKey, noLock)
+ }
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, 0)
+}
+
+// BulkRemove removes a list of objects from the cache
+func (c *Cache) BulkRemove(cacheKeys []string, noLock bool) {
+ for _, cacheKey := range cacheKeys {
+ c.Remove(cacheKey)
+ }
+}
+
+// Close is not used for Cache
+func (c *Cache) Close() error {
+ return nil
+}
+
+func (c *Cache) getFileName(cacheKey string) string {
+ prefix := strings.Replace(c.Config.Filesystem.CachePath+"/"+cacheKey+".", "//", "/", 1)
+ return prefix + "data"
+}
+
+// writeable returns true if the path is writeable by the calling process.
+func writeable(path string) bool {
+ return unix.Access(path, unix.W_OK) == nil
+}
+
+// makeDirectory creates a directory on the filesystem and exits the application in the event of a failure.
+func makeDirectory(path string) error {
+ err := os.MkdirAll(path, 0755)
+ if err != nil || !writeable(path) {
+ return fmt.Errorf("[%s] directory is not writeable by trickster: %v", path, err)
+ }
+
+ return nil
+}
diff --git a/internal/cache/filesystem/filesystem_test.go b/internal/cache/filesystem/filesystem_test.go
new file mode 100644
index 000000000..aa636ffd9
--- /dev/null
+++ b/internal/cache/filesystem/filesystem_test.go
@@ -0,0 +1,588 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package filesystem
+
+import (
+ "io/ioutil"
+ "os"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/util/log"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+const cacheType = "filesystem"
+const cacheKey = "cacheKey"
+
+func storeBenchmark(b *testing.B) Cache {
+ log.Logger = log.ConsoleLogger("none")
+ dir, _ := ioutil.TempDir("/tmp", cacheType)
+ cacheConfig := config.CachingConfig{CacheType: cacheType, Filesystem: config.FilesystemCacheConfig{CachePath: dir}, Index: config.CacheIndexConfig{ReapInterval: time.Second}}
+ fc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.BBolt.Filename)
+
+ err := fc.Connect()
+ if err != nil {
+ b.Error(err)
+ }
+
+ // it should store a value
+ for n := 0; n < b.N; n++ {
+ err = fc.Store(cacheKey+strconv.Itoa(n), []byte("data"+strconv.Itoa(n)), time.Duration(60)*time.Second)
+ if err != nil {
+ b.Error(err)
+ }
+ }
+ return fc
+}
+
+func newCacheConfig(t *testing.T) config.CachingConfig {
+ dir, err := ioutil.TempDir("/tmp", cacheType)
+ if err != nil {
+ t.Fatalf("could not create temp directory (%s): %s", dir, err)
+ }
+ return config.CachingConfig{CacheType: cacheType, Filesystem: config.FilesystemCacheConfig{CachePath: dir}, Index: config.CacheIndexConfig{ReapInterval: time.Second}}
+}
+
+func TestConfiguration(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+ cfg := fc.Configuration()
+ if cfg.CacheType != cacheType {
+ t.Fatalf("expected %s got %s", cacheType, cfg.CacheType)
+ }
+}
+
+func TestFilesystemCache_Connect(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+
+ // it should connect
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestFilesystemCache_ConnectFailed(t *testing.T) {
+ const expected = `[/root/noaccess.trickster.filesystem.cache] directory is not writeable by trickster:`
+ cacheConfig := newCacheConfig(t)
+ cacheConfig.Filesystem.CachePath = "/root/noaccess.trickster.filesystem.cache"
+ fc := Cache{Config: &cacheConfig}
+ // it should connect
+ err := fc.Connect()
+ if err == nil {
+ t.Errorf("expected error for %s", expected)
+ fc.Close()
+ }
+ if !strings.HasPrefix(err.Error(), expected) {
+ t.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+}
+
+func TestFilesystemCache_Store(t *testing.T) {
+
+ const expected1 = "invalid ttl: -1"
+ const expected2 = "open /root/noaccess.trickster.filesystem.cache/cacheKey.data:"
+
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should store a value
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should return an error
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(-1)*time.Second)
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ }
+ if err.Error() != expected1 {
+ t.Errorf("expected error '%s' got '%s'", expected1, err.Error())
+ }
+
+ cacheConfig.Filesystem.CachePath = "/root/noaccess.trickster.filesystem.cache"
+ // it should return an error
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err == nil {
+ t.Errorf("expected error for %s", expected2)
+ }
+ if !strings.HasPrefix(err.Error(), expected2) {
+ t.Errorf("expected error '%s' got '%s'", expected2, err.Error())
+ }
+
+}
+
+func BenchmarkCache_Store(b *testing.B) {
+ fc := storeBenchmark(b)
+ defer fc.Close()
+}
+
+func TestFilesystemCache_StoreNoIndex(t *testing.T) {
+
+ const expected = "value for key [] not in cache"
+
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer fc.Close()
+
+ // it should store a value
+ fc.storeNoIndex(cacheKey, []byte("data"))
+
+ // it should retrieve a value
+ data, ls, err := fc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // test for error when bad key name
+ fc.storeNoIndex("", []byte("data"))
+
+ data, ls, err = fc.retrieve("", false, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected)
+ fc.Close()
+ }
+ if err.Error() != expected {
+ t.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+ if string(data) != "" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_StoreNoIndex(b *testing.B) {
+ fc := storeBenchmark(b)
+ defer fc.Close()
+ for n := 0; n < b.N; n++ {
+ expected := `value for key [] not in cache`
+ // it should store a value
+ fc.storeNoIndex(cacheKey+strconv.Itoa(n), []byte("data"+strconv.Itoa(n)))
+
+ // it should retrieve a value
+ data, ls, err := fc.retrieve(cacheKey+strconv.Itoa(n), false, false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ // test for error when bad key name
+ fc.storeNoIndex("", []byte("data"+strconv.Itoa(n)))
+
+ data, ls, err = fc.retrieve("", false, false)
+ if err == nil {
+ b.Errorf("expected error for %s", expected)
+ fc.Close()
+ }
+ if err.Error() != expected {
+ b.Errorf("expected error '%s' got '%s'", expected, err.Error())
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestFilesystemCache_SetTTL(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ fc := Cache{Config: &cacheConfig}
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer fc.Close()
+
+ exp1 := fc.Index.GetExpiration(cacheKey)
+ if !exp1.IsZero() {
+ t.Errorf("expected Zero time, got %v", exp1)
+ }
+
+ // it should store a value
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ exp1 = fc.Index.GetExpiration(cacheKey)
+ if exp1.IsZero() {
+ t.Errorf("expected time %d, got zero", int(time.Now().Unix())+60)
+ }
+
+ e1 := int(exp1.Unix())
+
+ fc.SetTTL(cacheKey, time.Duration(3600)*time.Second)
+
+ exp2 := fc.Index.GetExpiration(cacheKey)
+ if exp2.IsZero() {
+ t.Errorf("expected time %d, got zero", int(time.Now().Unix())+3600)
+ }
+ e2 := int(exp2.Unix())
+
+ // should be around 3595
+ diff := e2 - e1
+ const expected = 3500
+
+ if diff < expected {
+ t.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, e2, e1)
+ }
+
+}
+
+func BenchmarkCache_SetTTL(b *testing.B) {
+ fc := storeBenchmark(b)
+ defer fc.Close()
+ for n := 0; n < b.N; n++ {
+ exp1 := fc.Index.GetExpiration(cacheKey + strconv.Itoa(n))
+ if exp1.IsZero() {
+ b.Errorf("expected time %d, got zero", int(time.Now().Unix())+60)
+ }
+
+ e1 := int(exp1.Unix())
+
+ fc.SetTTL(cacheKey+strconv.Itoa(n), time.Duration(3600)*time.Second)
+
+ exp2 := fc.Index.GetExpiration(cacheKey + strconv.Itoa(n))
+ if exp2.IsZero() {
+ b.Errorf("expected time %d, got zero", int(time.Now().Unix())+3600)
+ }
+ e2 := int(exp2.Unix())
+
+ // should be around 3595
+ diff := e2 - e1
+ const expected = 3500
+
+ if diff < expected {
+ b.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, e2, e1)
+ }
+ }
+}
+
+func TestFilesystemCache_Retrieve(t *testing.T) {
+
+ const expected1 = `value for key [cacheKey] not in cache`
+ const expected2 = `value for key [cacheKey] could not be deserialized from cache`
+
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := fc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // expire the object
+ fc.SetTTL(cacheKey, -1*time.Hour)
+
+ // this should now return error
+ data, ls, err = fc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ fc.Close()
+ }
+ if err.Error() != expected1 {
+ t.Errorf("expected error '%s' got '%s'", expected1, err.Error())
+ }
+ if string(data) != "" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ // should fail
+ filename := fc.getFileName(cacheKey)
+ err = ioutil.WriteFile(filename, []byte("junk"), os.FileMode(0777))
+ if err != nil {
+ t.Error(err)
+ }
+ _, ls, err = fc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected2)
+ }
+ if err.Error() != expected2 {
+ t.Errorf("expected error '%s' got '%s'", expected2, err.Error())
+ }
+ if ls != status.LookupStatusError {
+ t.Errorf("expected %s got %s", status.LookupStatusError, ls)
+ }
+}
+
+func BenchmarkCache_Retrieve(b *testing.B) {
+ fc := storeBenchmark(b)
+ defer fc.Close()
+
+ for n := 0; n < b.N; n++ {
+ expected1 := `value for key [` + cacheKey + strconv.Itoa(n) + `] not in cache`
+ expected2 := `value for key [` + cacheKey + strconv.Itoa(n) + `] could not be deserialized from cache`
+
+ data, ls, err := fc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // expire the object
+ fc.SetTTL(cacheKey+strconv.Itoa(n), -1*time.Hour)
+
+ // this should now return error
+ data, ls, err = fc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected error for %s", expected1)
+ fc.Close()
+ }
+ if err.Error() != expected1 {
+ b.Errorf("expected error '%s' got '%s'", expected1, err.Error())
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ filename := fc.getFileName(cacheKey + strconv.Itoa(n))
+ // create a corrupted cache entry and expect an error
+ ioutil.WriteFile(filename, []byte("junk"), os.FileMode(0777))
+
+ // it should fail to retrieve a value
+ data, ls, err = fc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected error for %s", expected2)
+ fc.Close()
+ }
+ if err.Error() != expected2 {
+ b.Errorf("expected error '%s' got '%s'", expected2, err.Error())
+ }
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestFilesystemCache_Remove(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer fc.Close()
+
+ // it should store a value
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := fc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ fc.Remove(cacheKey)
+
+ // it should be a cache miss
+ _, ls, err = fc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_Remove(b *testing.B) {
+ fc := storeBenchmark(b)
+ defer fc.Close()
+
+ for n := 0; n < b.N; n++ {
+ var data []byte
+ data, ls, err := fc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\"", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ fc.Remove(cacheKey + strconv.Itoa(n))
+
+ // this should now return error
+ data, ls, err = fc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ expectederr := `value for key [` + cacheKey + strconv.Itoa(n) + `] not in cache`
+ if err == nil {
+ b.Errorf("expected error for %s", expectederr)
+ fc.Close()
+ }
+ if err.Error() != expectederr {
+ b.Errorf("expected error '%s' got '%s'", expectederr, err.Error())
+ }
+
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestFilesystemCache_BulkRemove(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ defer os.RemoveAll(cacheConfig.Filesystem.CachePath)
+ fc := Cache{Config: &cacheConfig}
+
+ err := fc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer fc.Close()
+
+ // it should store a value
+ err = fc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := fc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ fc.BulkRemove([]string{cacheKey}, true)
+
+ // it should be a cache miss
+ _, ls, err = fc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_BulkRemove(b *testing.B) {
+ fc := storeBenchmark(b)
+ defer fc.Close()
+
+ var keyArray []string
+ for n := 0; n < b.N; n++ {
+ keyArray = append(keyArray, cacheKey+strconv.Itoa(n))
+ }
+
+ fc.BulkRemove(keyArray, true)
+
+ // it should be a cache miss
+ for n := 0; n < b.N; n++ {
+ _, ls, err := fc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
diff --git a/internal/cache/index/index.go b/internal/cache/index/index.go
new file mode 100644
index 000000000..806f1de7a
--- /dev/null
+++ b/internal/cache/index/index.go
@@ -0,0 +1,364 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package index
+
+import (
+ "sort"
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+//go:generate msgp
+
+// IndexKey is the key under which the index will write itself to its associated cache
+const IndexKey = "cache.index"
+
+var indexLock = sync.Mutex{}
+
+// Index maintains metadata about a Cache when Retention enforcement is managed internally,
+// like memory or bbolt. It is not used for independently managed caches like Redis.
+type Index struct {
+ // CacheSize represents the size of the cache in bytes
+ CacheSize int64 `msg:"cache_size"`
+ // ObjectCount represents the count of objects in the Cache
+ ObjectCount int64 `msg:"object_count"`
+ // Objects is a map of Objects in the Cache
+ Objects map[string]*Object `msg:"objects"`
+
+ name string `msg:"-"`
+ cacheType string `msg:"-"`
+ config config.CacheIndexConfig `msg:"-"`
+ bulkRemoveFunc func([]string, bool) `msg:"-"`
+ reapInterval time.Duration `msg:"-"`
+ flushInterval time.Duration `msg:"-"`
+ flushFunc func(cacheKey string, data []byte) `msg:"-"`
+ lastWrite time.Time `msg:"-"`
+}
+
+// ToBytes returns a serialized byte slice representing the Index
+func (idx *Index) ToBytes() []byte {
+ bytes, _ := idx.MarshalMsg(nil)
+ return bytes
+}
+
+// Object contains metadata about an item in the Cache
+type Object struct {
+ // Key represents the name of the Object and is the accessor in a hashed collection of Cache Objects
+ Key string `msg:"key"`
+ // Expiration represents the time that the Object expires from Cache
+ Expiration time.Time `msg:"expiration"`
+ // LastWrite is the time the object was last Written
+ LastWrite time.Time `msg:"lastwrite"`
+ // LastAccess is the time the object was last Accessed
+ LastAccess time.Time `msg:"lastaccess"`
+ // Size the size of the Object in bytes
+ Size int64 `msg:"size"`
+ // Value is the value of the Object stored in the Cache
+ // It is used by Caches but not by the Index
+ Value []byte `msg:"value,omitempty"`
+ // DirectValue is an interface value for storing objects by reference to a memory cache
+ // Since we'd never recover a memory cache index from memory on startup, no need to msgpk
+ ReferenceValue cache.ReferenceObject `msg:"-"`
+}
+
+// ToBytes returns a serialized byte slice representing the Object
+func (o *Object) ToBytes() []byte {
+ bytes, _ := o.MarshalMsg(nil)
+ return bytes
+}
+
+// ObjectFromBytes returns a deserialized Cache Object from a seralized byte slice
+func ObjectFromBytes(data []byte) (*Object, error) {
+ o := &Object{}
+ _, err := o.UnmarshalMsg(data)
+ return o, err
+}
+
+// NewIndex returns a new Index based on the provided inputs
+func NewIndex(cacheName, cacheType string, indexData []byte, cfg config.CacheIndexConfig, bulkRemoveFunc func([]string, bool), flushFunc func(cacheKey string, data []byte)) *Index {
+ i := &Index{}
+
+ if len(indexData) > 0 {
+ i.UnmarshalMsg(indexData)
+ } else {
+ i.Objects = make(map[string]*Object)
+ }
+
+ i.name = cacheName
+ i.cacheType = cacheType
+ i.flushInterval = cfg.FlushInterval
+ i.flushFunc = flushFunc
+ i.reapInterval = cfg.ReapInterval
+ i.bulkRemoveFunc = bulkRemoveFunc
+ i.config = cfg
+
+ if flushFunc != nil {
+ if i.flushInterval > 0 {
+ go i.flusher()
+ } else {
+ log.Warn("cache index flusher did not start", log.Pairs{"cacheName": i.name, "flushInterval": i.flushInterval})
+ }
+ }
+
+ if i.reapInterval > 0 {
+ go i.reaper()
+ } else {
+ log.Warn("cache reaper did not start", log.Pairs{"cacheName": i.name, "reapInterval": i.reapInterval})
+ }
+
+ metrics.CacheMaxObjects.WithLabelValues(cacheName, cacheType).Set(float64(cfg.MaxSizeObjects))
+ metrics.CacheMaxBytes.WithLabelValues(cacheName, cacheType).Set(float64(cfg.MaxSizeBytes))
+
+ return i
+}
+
+// UpdateObjectAccessTime updates the LastAccess for the object with the provided key
+func (idx *Index) UpdateObjectAccessTime(key string) {
+ indexLock.Lock()
+ if _, ok := idx.Objects[key]; ok {
+ idx.Objects[key].LastAccess = time.Now()
+ }
+ indexLock.Unlock()
+
+}
+
+// UpdateObjectTTL updates the Expiration for the object with the provided key
+func (idx *Index) UpdateObjectTTL(key string, ttl time.Duration) {
+ indexLock.Lock()
+ if _, ok := idx.Objects[key]; ok {
+ idx.Objects[key].Expiration = time.Now().Add(ttl)
+ }
+ indexLock.Unlock()
+}
+
+// UpdateObject writes or updates the Index Metadata for the provided Object
+func (idx *Index) UpdateObject(obj *Object) {
+
+ key := obj.Key
+ if key == "" {
+ return
+ }
+
+ indexLock.Lock()
+
+ idx.lastWrite = time.Now()
+
+ if obj.ReferenceValue != nil {
+ obj.Size = int64(obj.ReferenceValue.Size())
+ } else {
+ obj.Size = int64(len(obj.Value))
+ }
+ obj.Value = nil
+ obj.LastAccess = time.Now()
+ obj.LastWrite = obj.LastAccess
+
+ if o, ok := idx.Objects[key]; ok {
+ idx.CacheSize += o.Size - idx.Objects[key].Size
+ } else {
+ idx.CacheSize += obj.Size
+ idx.ObjectCount++
+ }
+
+ cache.ObserveCacheSizeChange(idx.name, idx.cacheType, idx.CacheSize, idx.ObjectCount)
+
+ idx.Objects[key] = obj
+ indexLock.Unlock()
+}
+
+// RemoveObject removes an Object's Metadata from the Index
+func (idx *Index) RemoveObject(key string, noLock bool) {
+
+ if !noLock {
+ indexLock.Lock()
+ idx.lastWrite = time.Now()
+ }
+ if o, ok := idx.Objects[key]; ok {
+ idx.CacheSize -= o.Size
+ idx.ObjectCount--
+
+ cache.ObserveCacheOperation(idx.name, idx.cacheType, "del", "none", float64(o.Size))
+
+ delete(idx.Objects, key)
+ cache.ObserveCacheSizeChange(idx.name, idx.cacheType, idx.CacheSize, idx.ObjectCount)
+ }
+ if !noLock {
+ indexLock.Unlock()
+ }
+
+}
+
+// GetExpiration returns the cache index's expiration for the object of the given key
+func (idx *Index) GetExpiration(cacheKey string) time.Time {
+ indexLock.Lock()
+ if o, ok := idx.Objects[cacheKey]; ok {
+ indexLock.Unlock()
+ return o.Expiration
+ }
+ indexLock.Unlock()
+ return time.Time{}
+}
+
+// flusher periodically calls the cache's index flush func that writes the cache index to disk
+func (idx *Index) flusher() {
+ var lastFlush time.Time
+ for {
+ time.Sleep(idx.flushInterval)
+ if idx.lastWrite.Before(lastFlush) {
+ continue
+ }
+ idx.flushOnce()
+ lastFlush = time.Now()
+ }
+}
+
+func (idx *Index) flushOnce() {
+ indexLock.Lock()
+ bytes, err := idx.MarshalMsg(nil)
+ indexLock.Unlock()
+ if err != nil {
+ log.Warn("unable to serialize index for flushing", log.Pairs{"cacheName": idx.name, "detail": err.Error()})
+ return
+ }
+ idx.flushFunc(IndexKey, bytes)
+}
+
+// reaper continually iterates through the cache to find expired elements and removes them
+func (idx *Index) reaper() {
+ for {
+ idx.reap()
+ time.Sleep(idx.reapInterval)
+ }
+}
+
+type objectsAtime []*Object
+
+// reap makes a single iteration through the cache index to to find and remove expired elements
+// and evict least-recently-accessed elements to maintain the Maximum allowed Cache Size
+func (idx *Index) reap() {
+
+ indexLock.Lock()
+ defer indexLock.Unlock()
+
+ removals := make([]string, 0)
+ remainders := make(objectsAtime, 0, idx.ObjectCount)
+
+ var cacheChanged bool
+
+ now := time.Now()
+
+ for _, o := range idx.Objects {
+ if o.Key == IndexKey {
+ continue
+ }
+ if o.Expiration.Before(now) && !o.Expiration.IsZero() {
+ removals = append(removals, o.Key)
+ } else {
+ remainders = append(remainders, o)
+ }
+ }
+
+ if len(removals) > 0 {
+ cache.ObserveCacheEvent(idx.name, idx.cacheType, "eviction", "ttl")
+ idx.bulkRemoveFunc(removals, true)
+ cacheChanged = true
+ }
+
+ if ((idx.config.MaxSizeBytes > 0 && idx.CacheSize > idx.config.MaxSizeBytes) || (idx.config.MaxSizeObjects > 0 && idx.ObjectCount > idx.config.MaxSizeObjects)) && len(remainders) > 0 {
+
+ var evictionType string
+ if idx.config.MaxSizeBytes > 0 && idx.CacheSize > idx.config.MaxSizeBytes {
+ evictionType = "size_bytes"
+ } else if idx.config.MaxSizeObjects > 0 && idx.ObjectCount > idx.config.MaxSizeObjects {
+ evictionType = "size_objects"
+ } else {
+ return
+ }
+
+ log.Debug("max cache size reached. evicting least-recently-accessed records",
+ log.Pairs{
+ "reason": evictionType,
+ "cacheSizeBytes": idx.CacheSize, "maxSizeBytes": idx.config.MaxSizeBytes,
+ "cacheSizeObjects": idx.ObjectCount, "maxSizeObjects": idx.config.MaxSizeObjects,
+ },
+ )
+
+ removals = make([]string, 0)
+
+ sort.Sort(remainders)
+
+ i := 0
+ j := len(remainders)
+
+ if evictionType == "size_bytes" {
+ bytesNeeded := (idx.CacheSize - idx.config.MaxSizeBytes)
+ if idx.config.MaxSizeBytes > idx.config.MaxSizeBackoffBytes {
+ bytesNeeded += idx.config.MaxSizeBackoffBytes
+ }
+ bytesSelected := int64(0)
+ for bytesSelected < bytesNeeded && i < j {
+ removals = append(removals, remainders[i].Key)
+ bytesSelected += remainders[i].Size
+ i++
+ }
+ } else {
+ objectsNeeded := (idx.ObjectCount - idx.config.MaxSizeObjects)
+ if idx.config.MaxSizeObjects > idx.config.MaxSizeBackoffObjects {
+ objectsNeeded += idx.config.MaxSizeBackoffObjects
+ }
+ objectsSelected := int64(0)
+ for objectsSelected < objectsNeeded && i < j {
+ removals = append(removals, remainders[i].Key)
+ objectsSelected++
+ i++
+ }
+ }
+
+ if len(removals) > 0 {
+ cache.ObserveCacheEvent(idx.name, idx.cacheType, "eviction", evictionType)
+ idx.bulkRemoveFunc(removals, true)
+ cacheChanged = true
+ }
+
+ log.Debug("size-based cache eviction exercise completed",
+ log.Pairs{
+ "reason": evictionType,
+ "cacheSizeBytes": idx.CacheSize, "maxSizeBytes": idx.config.MaxSizeBytes,
+ "cacheSizeObjects": idx.ObjectCount, "maxSizeObjects": idx.config.MaxSizeObjects,
+ })
+
+ }
+ if cacheChanged {
+ idx.lastWrite = time.Now()
+ }
+}
+
+// Len returns the length of an array of Prometheus model.Times
+func (o objectsAtime) Len() int {
+ return len(o)
+}
+
+// Less returns true if i comes before j
+func (o objectsAtime) Less(i, j int) bool {
+ return o[i].LastAccess.Before(o[j].LastAccess)
+}
+
+// Swap modifies an array by of Prometheus model.Times swapping the values in indexes i and j
+func (o objectsAtime) Swap(i, j int) {
+ o[i], o[j] = o[j], o[i]
+}
diff --git a/internal/cache/index/index_gen.go b/internal/cache/index/index_gen.go
new file mode 100644
index 000000000..46b6217a5
--- /dev/null
+++ b/internal/cache/index/index_gen.go
@@ -0,0 +1,467 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package index
+
+// Code generated by github.com/tinylib/msgp DO NOT EDIT.
+
+import (
+ "github.com/tinylib/msgp/msgp"
+)
+
+// DecodeMsg implements msgp.Decodable
+func (z *Index) DecodeMsg(dc *msgp.Reader) (err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, err = dc.ReadMapKeyPtr()
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "CacheSize":
+ z.CacheSize, err = dc.ReadInt64()
+ if err != nil {
+ return
+ }
+ case "ObjectCount":
+ z.ObjectCount, err = dc.ReadInt64()
+ if err != nil {
+ return
+ }
+ case "Objects":
+ var zb0002 uint32
+ zb0002, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ if z.Objects == nil {
+ z.Objects = make(map[string]*Object, zb0002)
+ } else if len(z.Objects) > 0 {
+ for key := range z.Objects {
+ delete(z.Objects, key)
+ }
+ }
+ for zb0002 > 0 {
+ zb0002--
+ var za0001 string
+ var za0002 *Object
+ za0001, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ if dc.IsNil() {
+ err = dc.ReadNil()
+ if err != nil {
+ return
+ }
+ za0002 = nil
+ } else {
+ if za0002 == nil {
+ za0002 = new(Object)
+ }
+ err = za0002.DecodeMsg(dc)
+ if err != nil {
+ return
+ }
+ }
+ z.Objects[za0001] = za0002
+ }
+ default:
+ err = dc.Skip()
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// EncodeMsg implements msgp.Encodable
+func (z *Index) EncodeMsg(en *msgp.Writer) (err error) {
+ // map header, size 3
+ // write "CacheSize"
+ err = en.Append(0x83, 0xa9, 0x43, 0x61, 0x63, 0x68, 0x65, 0x53, 0x69, 0x7a, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteInt64(z.CacheSize)
+ if err != nil {
+ return
+ }
+ // write "ObjectCount"
+ err = en.Append(0xab, 0x4f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x43, 0x6f, 0x75, 0x6e, 0x74)
+ if err != nil {
+ return
+ }
+ err = en.WriteInt64(z.ObjectCount)
+ if err != nil {
+ return
+ }
+ // write "Objects"
+ err = en.Append(0xa7, 0x4f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73)
+ if err != nil {
+ return
+ }
+ err = en.WriteMapHeader(uint32(len(z.Objects)))
+ if err != nil {
+ return
+ }
+ for za0001, za0002 := range z.Objects {
+ err = en.WriteString(za0001)
+ if err != nil {
+ return
+ }
+ if za0002 == nil {
+ err = en.WriteNil()
+ if err != nil {
+ return
+ }
+ } else {
+ err = za0002.EncodeMsg(en)
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// MarshalMsg implements msgp.Marshaler
+func (z *Index) MarshalMsg(b []byte) (o []byte, err error) {
+ o = msgp.Require(b, z.Msgsize())
+ // map header, size 3
+ // string "CacheSize"
+ o = append(o, 0x83, 0xa9, 0x43, 0x61, 0x63, 0x68, 0x65, 0x53, 0x69, 0x7a, 0x65)
+ o = msgp.AppendInt64(o, z.CacheSize)
+ // string "ObjectCount"
+ o = append(o, 0xab, 0x4f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x43, 0x6f, 0x75, 0x6e, 0x74)
+ o = msgp.AppendInt64(o, z.ObjectCount)
+ // string "Objects"
+ o = append(o, 0xa7, 0x4f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73)
+ o = msgp.AppendMapHeader(o, uint32(len(z.Objects)))
+ for za0001, za0002 := range z.Objects {
+ o = msgp.AppendString(o, za0001)
+ if za0002 == nil {
+ o = msgp.AppendNil(o)
+ } else {
+ o, err = za0002.MarshalMsg(o)
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// UnmarshalMsg implements msgp.Unmarshaler
+func (z *Index) UnmarshalMsg(bts []byte) (o []byte, err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, bts, err = msgp.ReadMapKeyZC(bts)
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "CacheSize":
+ z.CacheSize, bts, err = msgp.ReadInt64Bytes(bts)
+ if err != nil {
+ return
+ }
+ case "ObjectCount":
+ z.ObjectCount, bts, err = msgp.ReadInt64Bytes(bts)
+ if err != nil {
+ return
+ }
+ case "Objects":
+ var zb0002 uint32
+ zb0002, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ if z.Objects == nil {
+ z.Objects = make(map[string]*Object, zb0002)
+ } else if len(z.Objects) > 0 {
+ for key := range z.Objects {
+ delete(z.Objects, key)
+ }
+ }
+ for zb0002 > 0 {
+ var za0001 string
+ var za0002 *Object
+ zb0002--
+ za0001, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ if msgp.IsNil(bts) {
+ bts, err = msgp.ReadNilBytes(bts)
+ if err != nil {
+ return
+ }
+ za0002 = nil
+ } else {
+ if za0002 == nil {
+ za0002 = new(Object)
+ }
+ bts, err = za0002.UnmarshalMsg(bts)
+ if err != nil {
+ return
+ }
+ }
+ z.Objects[za0001] = za0002
+ }
+ default:
+ bts, err = msgp.Skip(bts)
+ if err != nil {
+ return
+ }
+ }
+ }
+ o = bts
+ return
+}
+
+// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
+func (z *Index) Msgsize() (s int) {
+ s = 1 + 10 + msgp.Int64Size + 12 + msgp.Int64Size + 8 + msgp.MapHeaderSize
+ if z.Objects != nil {
+ for za0001, za0002 := range z.Objects {
+ _ = za0002
+ s += msgp.StringPrefixSize + len(za0001)
+ if za0002 == nil {
+ s += msgp.NilSize
+ } else {
+ s += za0002.Msgsize()
+ }
+ }
+ }
+ return
+}
+
+// DecodeMsg implements msgp.Decodable
+func (z *Object) DecodeMsg(dc *msgp.Reader) (err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, err = dc.ReadMapKeyPtr()
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "key":
+ z.Key, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ case "expiration":
+ z.Expiration, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "lastwrite":
+ z.LastWrite, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "lastaccess":
+ z.LastAccess, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "size":
+ z.Size, err = dc.ReadInt64()
+ if err != nil {
+ return
+ }
+ case "value":
+ z.Value, err = dc.ReadBytes(z.Value)
+ if err != nil {
+ return
+ }
+ default:
+ err = dc.Skip()
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// EncodeMsg implements msgp.Encodable
+func (z *Object) EncodeMsg(en *msgp.Writer) (err error) {
+ // map header, size 6
+ // write "key"
+ err = en.Append(0x86, 0xa3, 0x6b, 0x65, 0x79)
+ if err != nil {
+ return
+ }
+ err = en.WriteString(z.Key)
+ if err != nil {
+ return
+ }
+ // write "expiration"
+ err = en.Append(0xaa, 0x65, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.Expiration)
+ if err != nil {
+ return
+ }
+ // write "lastwrite"
+ err = en.Append(0xa9, 0x6c, 0x61, 0x73, 0x74, 0x77, 0x72, 0x69, 0x74, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.LastWrite)
+ if err != nil {
+ return
+ }
+ // write "lastaccess"
+ err = en.Append(0xaa, 0x6c, 0x61, 0x73, 0x74, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.LastAccess)
+ if err != nil {
+ return
+ }
+ // write "size"
+ err = en.Append(0xa4, 0x73, 0x69, 0x7a, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteInt64(z.Size)
+ if err != nil {
+ return
+ }
+ // write "value"
+ err = en.Append(0xa5, 0x76, 0x61, 0x6c, 0x75, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteBytes(z.Value)
+ if err != nil {
+ return
+ }
+ return
+}
+
+// MarshalMsg implements msgp.Marshaler
+func (z *Object) MarshalMsg(b []byte) (o []byte, err error) {
+ o = msgp.Require(b, z.Msgsize())
+ // map header, size 6
+ // string "key"
+ o = append(o, 0x86, 0xa3, 0x6b, 0x65, 0x79)
+ o = msgp.AppendString(o, z.Key)
+ // string "expiration"
+ o = append(o, 0xaa, 0x65, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e)
+ o = msgp.AppendTime(o, z.Expiration)
+ // string "lastwrite"
+ o = append(o, 0xa9, 0x6c, 0x61, 0x73, 0x74, 0x77, 0x72, 0x69, 0x74, 0x65)
+ o = msgp.AppendTime(o, z.LastWrite)
+ // string "lastaccess"
+ o = append(o, 0xaa, 0x6c, 0x61, 0x73, 0x74, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73)
+ o = msgp.AppendTime(o, z.LastAccess)
+ // string "size"
+ o = append(o, 0xa4, 0x73, 0x69, 0x7a, 0x65)
+ o = msgp.AppendInt64(o, z.Size)
+ // string "value"
+ o = append(o, 0xa5, 0x76, 0x61, 0x6c, 0x75, 0x65)
+ o = msgp.AppendBytes(o, z.Value)
+ return
+}
+
+// UnmarshalMsg implements msgp.Unmarshaler
+func (z *Object) UnmarshalMsg(bts []byte) (o []byte, err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, bts, err = msgp.ReadMapKeyZC(bts)
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "key":
+ z.Key, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ case "expiration":
+ z.Expiration, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "lastwrite":
+ z.LastWrite, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "lastaccess":
+ z.LastAccess, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "size":
+ z.Size, bts, err = msgp.ReadInt64Bytes(bts)
+ if err != nil {
+ return
+ }
+ case "value":
+ z.Value, bts, err = msgp.ReadBytesBytes(bts, z.Value)
+ if err != nil {
+ return
+ }
+ default:
+ bts, err = msgp.Skip(bts)
+ if err != nil {
+ return
+ }
+ }
+ }
+ o = bts
+ return
+}
+
+// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
+func (z *Object) Msgsize() (s int) {
+ s = 1 + 4 + msgp.StringPrefixSize + len(z.Key) + 11 + msgp.TimeSize + 10 + msgp.TimeSize + 11 + msgp.TimeSize + 5 + msgp.Int64Size + 6 + msgp.BytesPrefixSize + len(z.Value)
+ return
+}
diff --git a/internal/cache/index/index_gen_test.go b/internal/cache/index/index_gen_test.go
new file mode 100644
index 000000000..bd932c02b
--- /dev/null
+++ b/internal/cache/index/index_gen_test.go
@@ -0,0 +1,249 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package index
+
+// Code generated by github.com/tinylib/msgp DO NOT EDIT.
+
+import (
+ "bytes"
+ "testing"
+
+ "github.com/tinylib/msgp/msgp"
+)
+
+func TestMarshalUnmarshalIndex(t *testing.T) {
+ v := Index{}
+ bts, err := v.MarshalMsg(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ left, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
+ }
+
+ left, err = msgp.Skip(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
+ }
+}
+
+func BenchmarkMarshalMsgIndex(b *testing.B) {
+ v := Index{}
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.MarshalMsg(nil)
+ }
+}
+
+func BenchmarkAppendMsgIndex(b *testing.B) {
+ v := Index{}
+ bts := make([]byte, 0, v.Msgsize())
+ bts, _ = v.MarshalMsg(bts[0:0])
+ b.SetBytes(int64(len(bts)))
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ bts, _ = v.MarshalMsg(bts[0:0])
+ }
+}
+
+func BenchmarkUnmarshalIndex(b *testing.B) {
+ v := Index{}
+ bts, _ := v.MarshalMsg(nil)
+ b.ReportAllocs()
+ b.SetBytes(int64(len(bts)))
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ _, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
+
+func TestEncodeDecodeIndex(t *testing.T) {
+ v := Index{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+
+ m := v.Msgsize()
+ if buf.Len() > m {
+ t.Logf("WARNING: Msgsize() for %v is inaccurate", v)
+ }
+
+ vn := Index{}
+ err := msgp.Decode(&buf, &vn)
+ if err != nil {
+ t.Error(err)
+ }
+
+ buf.Reset()
+ msgp.Encode(&buf, &v)
+ err = msgp.NewReader(&buf).Skip()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkEncodeIndex(b *testing.B) {
+ v := Index{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ en := msgp.NewWriter(msgp.Nowhere)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.EncodeMsg(en)
+ }
+ en.Flush()
+}
+
+func BenchmarkDecodeIndex(b *testing.B) {
+ v := Index{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ rd := msgp.NewEndlessReader(buf.Bytes(), b)
+ dc := msgp.NewReader(rd)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ err := v.DecodeMsg(dc)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
+
+func TestMarshalUnmarshalObject(t *testing.T) {
+ v := Object{}
+ bts, err := v.MarshalMsg(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ left, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
+ }
+
+ left, err = msgp.Skip(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
+ }
+}
+
+func BenchmarkMarshalMsgObject(b *testing.B) {
+ v := Object{}
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.MarshalMsg(nil)
+ }
+}
+
+func BenchmarkAppendMsgObject(b *testing.B) {
+ v := Object{}
+ bts := make([]byte, 0, v.Msgsize())
+ bts, _ = v.MarshalMsg(bts[0:0])
+ b.SetBytes(int64(len(bts)))
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ bts, _ = v.MarshalMsg(bts[0:0])
+ }
+}
+
+func BenchmarkUnmarshalObject(b *testing.B) {
+ v := Object{}
+ bts, _ := v.MarshalMsg(nil)
+ b.ReportAllocs()
+ b.SetBytes(int64(len(bts)))
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ _, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
+
+func TestEncodeDecodeObject(t *testing.T) {
+ v := Object{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+
+ m := v.Msgsize()
+ if buf.Len() > m {
+ t.Logf("WARNING: Msgsize() for %v is inaccurate", v)
+ }
+
+ vn := Object{}
+ err := msgp.Decode(&buf, &vn)
+ if err != nil {
+ t.Error(err)
+ }
+
+ buf.Reset()
+ msgp.Encode(&buf, &v)
+ err = msgp.NewReader(&buf).Skip()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkEncodeObject(b *testing.B) {
+ v := Object{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ en := msgp.NewWriter(msgp.Nowhere)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.EncodeMsg(en)
+ }
+ en.Flush()
+}
+
+func BenchmarkDecodeObject(b *testing.B) {
+ v := Object{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ rd := msgp.NewEndlessReader(buf.Bytes(), b)
+ dc := msgp.NewReader(rd)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ err := v.DecodeMsg(dc)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
diff --git a/internal/cache/index/index_test.go b/internal/cache/index/index_test.go
new file mode 100644
index 000000000..ea0f451fc
--- /dev/null
+++ b/internal/cache/index/index_test.go
@@ -0,0 +1,286 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package index
+
+import (
+ "sort"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+var testBulkIndex *Index
+
+func testBulkRemoveFunc(cacheKeys []string, noLock bool) {
+ for _, cacheKey := range cacheKeys {
+ testBulkIndex.RemoveObject(cacheKey, noLock)
+ }
+}
+func fakeFlusherFunc(string, []byte) {}
+
+type testReferenceObject struct {
+}
+
+func (r *testReferenceObject) Size() int {
+ return 1
+}
+
+func TestNewIndex(t *testing.T) {
+ cacheConfig := &config.CachingConfig{CacheType: "test", Index: config.CacheIndexConfig{ReapInterval: time.Second * time.Duration(10), FlushInterval: time.Second * time.Duration(10)}}
+ idx := NewIndex("test", "test", nil, cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+
+ // this gives a chance for the reaper to run through for test coverage
+ time.Sleep(1 * time.Second)
+
+ if idx.name != "test" {
+ t.Errorf("expected test got %s", idx.name)
+ }
+
+ idx.flushOnce()
+
+ idx2 := NewIndex("test", "test", idx.ToBytes(), cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+ if idx2 == nil {
+ t.Errorf("nil cache index")
+ }
+
+ cacheConfig.Index.FlushInterval = 0
+ cacheConfig.Index.ReapInterval = 0
+ idx3 := NewIndex("test", "test", nil, cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+ if idx3 == nil {
+ t.Errorf("nil cache index")
+ }
+
+}
+
+func TestReap(t *testing.T) {
+
+ cacheConfig := &config.CachingConfig{CacheType: "test", Index: config.CacheIndexConfig{ReapInterval: time.Second * time.Duration(10), FlushInterval: time.Second * time.Duration(10)}}
+ cacheConfig.Index.MaxSizeObjects = 5
+ cacheConfig.Index.MaxSizeBackoffObjects = 3
+ cacheConfig.Index.MaxSizeBytes = 100
+ cacheConfig.Index.MaxSizeBackoffBytes = 30
+
+ idx := NewIndex("test", "test", nil, cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+ if idx.name != "test" {
+ t.Errorf("expected test got %s", idx.name)
+ }
+
+ testBulkIndex = idx
+
+ // add fake index key to cover the case that the reaper must skip it
+ idx.UpdateObject(&Object{Key: "cache.index", Value: []byte("test_value")})
+
+ // add expired key to cover the case that the reaper remove it
+ idx.UpdateObject(&Object{Key: "test.1", Value: []byte("test_value"), Expiration: time.Now().Add(-time.Minute)})
+
+ // add key with no expiration which should not be reaped
+ idx.UpdateObject(&Object{Key: "test.2", Value: []byte("test_value")})
+
+ // add key with future expiration which should not be reaped
+ idx.UpdateObject(&Object{Key: "test.3", Value: []byte("test_value"), Expiration: time.Now().Add(time.Minute)})
+
+ // trigger a reap that will only remove expired elements but not size down the full cache
+ idx.reap()
+
+ // add key with future expiration which should not be reaped
+ idx.UpdateObject(&Object{Key: "test.4", Value: []byte("test_value"), Expiration: time.Now().Add(time.Minute)})
+
+ // add key with future expiration which should not be reaped
+ idx.UpdateObject(&Object{Key: "test.5", Value: []byte("test_value"), Expiration: time.Now().Add(time.Minute)})
+
+ // add key with future expiration which should not be reaped
+ idx.UpdateObject(&Object{Key: "test.6", Value: []byte("test_value"), Expiration: time.Now().Add(time.Minute)})
+
+ // trigger size-based reap eviction of some elements
+ idx.reap()
+
+ if _, ok := idx.Objects["test.1"]; ok {
+ t.Errorf("expected key %s to be missing", "test.1")
+ }
+
+ if _, ok := idx.Objects["test.2"]; ok {
+ t.Errorf("expected key %s to be missing", "test.2")
+ }
+
+ if _, ok := idx.Objects["test.3"]; ok {
+ t.Errorf("expected key %s to be missing", "test.3")
+ }
+
+ if _, ok := idx.Objects["test.4"]; ok {
+ t.Errorf("expected key %s to be missing", "test.4")
+ }
+
+ if _, ok := idx.Objects["test.5"]; ok {
+ t.Errorf("expected key %s to be missing", "test.5")
+ }
+
+ if _, ok := idx.Objects["test.6"]; !ok {
+ t.Errorf("expected key %s to be present", "test.6")
+ }
+
+ // add key with large body to reach byte size threshold
+ idx.UpdateObject(&Object{Key: "test.7", Value: []byte("test_value00000000000000000000000000000000000000000000000000000000000000000000000000000"), Expiration: time.Now().Add(time.Minute)})
+
+ // trigger a byte-based reap
+ idx.reap()
+
+ // only cache index should be left
+
+ if _, ok := idx.Objects["test.6"]; ok {
+ t.Errorf("expected key %s to be missing", "test.6")
+ }
+
+ if _, ok := idx.Objects["test.7"]; ok {
+ t.Errorf("expected key %s to be missing", "test.7")
+ }
+
+}
+
+func TestObjectFromBytes(t *testing.T) {
+
+ obj := &Object{}
+ b := obj.ToBytes()
+ obj2, err := ObjectFromBytes(b)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if obj2 == nil {
+ t.Errorf("nil cache index")
+ }
+
+}
+
+func TestUpdateObject(t *testing.T) {
+
+ obj := Object{Key: "", Value: []byte("test_value")}
+ cacheConfig := &config.CachingConfig{CacheType: "test", Index: config.CacheIndexConfig{ReapInterval: time.Second * time.Duration(10), FlushInterval: time.Second * time.Duration(10)}}
+ idx := NewIndex("test", "test", nil, cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+
+ idx.UpdateObject(&obj)
+ if _, ok := idx.Objects["test"]; ok {
+ t.Errorf("test object should be missing from index")
+ }
+
+ obj.Key = "test"
+
+ idx.UpdateObject(&obj)
+ if _, ok := idx.Objects["test"]; !ok {
+ t.Errorf("test object missing from index")
+ }
+
+ // do it again to cover the index hit case
+ idx.UpdateObject(&obj)
+ if _, ok := idx.Objects["test"]; !ok {
+ t.Errorf("test object missing from index")
+ }
+
+ idx.Objects["test"].LastAccess = time.Time{}
+ idx.UpdateObjectAccessTime("test")
+
+ if idx.Objects["test"].LastAccess.IsZero() {
+ t.Errorf("test object last access time is wrong")
+ }
+
+ obj = Object{Key: "test2", ReferenceValue: &testReferenceObject{}}
+
+ idx.UpdateObject(&obj)
+ if _, ok := idx.Objects["test2"]; !ok {
+ t.Errorf("test object missing from index")
+ }
+
+}
+
+func TestRemoveObject(t *testing.T) {
+
+ obj := Object{Key: "test", Value: []byte("test_value")}
+ cacheConfig := &config.CachingConfig{CacheType: "test", Index: config.CacheIndexConfig{ReapInterval: time.Second * time.Duration(10), FlushInterval: time.Second * time.Duration(10)}}
+ idx := NewIndex("test", "test", nil, cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+
+ idx.UpdateObject(&obj)
+ if _, ok := idx.Objects["test"]; !ok {
+ t.Errorf("test object missing from index")
+ }
+
+ idx.RemoveObject("test", false)
+ if _, ok := idx.Objects["test"]; ok {
+ t.Errorf("test object should be missing from index")
+ }
+
+}
+
+func TestSort(t *testing.T) {
+
+ o := objectsAtime{
+ &Object{
+ Key: "3",
+ LastAccess: time.Unix(3, 0),
+ },
+ &Object{
+ Key: "1",
+ LastAccess: time.Unix(1, 0),
+ },
+ &Object{
+ Key: "2",
+ LastAccess: time.Unix(2, 0),
+ },
+ }
+ sort.Sort(o)
+
+ if o[0].Key != "1" {
+ t.Errorf("expected %s got %s", "1", o[0].Key)
+ }
+
+ if o[1].Key != "2" {
+ t.Errorf("expected %s got %s", "2", o[1].Key)
+ }
+
+ if o[2].Key != "3" {
+ t.Errorf("expected %s got %s", "3", o[2].Key)
+ }
+
+}
+
+func TestUpdateObjectTTL(t *testing.T) {
+
+ cacheKey := "test-ttl-key"
+ obj := Object{Key: cacheKey, Value: []byte("test_value")}
+ cacheConfig := &config.CachingConfig{CacheType: "test", Index: config.CacheIndexConfig{ReapInterval: time.Second * time.Duration(10), FlushInterval: time.Second * time.Duration(10)}}
+ idx := NewIndex("test", "test", nil, cacheConfig.Index, testBulkRemoveFunc, fakeFlusherFunc)
+
+ exp := idx.GetExpiration(cacheKey)
+ if !exp.IsZero() {
+ t.Errorf("expected zero time, got %v", exp)
+ }
+
+ idx.UpdateObject(&obj)
+
+ idx.UpdateObjectTTL(cacheKey, time.Duration(3600)*time.Second)
+
+ if obj.Expiration.IsZero() {
+ t.Errorf("expected non-zero time, got %v", obj.Expiration)
+ }
+
+ exp = idx.GetExpiration(cacheKey)
+ if exp.IsZero() {
+ t.Errorf("expected non-zero time, got %v", obj.Expiration)
+ }
+
+}
diff --git a/internal/cache/memory/memory.go b/internal/cache/memory/memory.go
new file mode 100644
index 000000000..0e0e0860d
--- /dev/null
+++ b/internal/cache/memory/memory.go
@@ -0,0 +1,173 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package memory
+
+import (
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/index"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/pkg/locks"
+)
+
+var lockPrefix string
+
+// Cache defines a a Memory Cache client that conforms to the Cache interface
+type Cache struct {
+ Name string
+ client sync.Map
+ Config *config.CachingConfig
+ Index *index.Index
+}
+
+// Configuration returns the Configuration for the Cache object
+func (c *Cache) Configuration() *config.CachingConfig {
+ return c.Config
+}
+
+// Connect initializes the Cache
+func (c *Cache) Connect() error {
+ log.Info("memorycache setup", log.Pairs{"name": c.Name, "maxSizeBytes": c.Config.Index.MaxSizeBytes, "maxSizeObjects": c.Config.Index.MaxSizeObjects})
+ lockPrefix = c.Name + ".memory."
+ c.client = sync.Map{}
+ c.Index = index.NewIndex(c.Name, c.Config.CacheType, nil, c.Config.Index, c.BulkRemove, nil)
+ return nil
+}
+
+// StoreReference stores an object directly to the memory cache without requiring serialization
+func (c *Cache) StoreReference(cacheKey string, data cache.ReferenceObject, ttl time.Duration) error {
+ return c.store(cacheKey, nil, data, ttl, true)
+}
+
+// Store places an object in the cache using the specified key and ttl
+func (c *Cache) Store(cacheKey string, data []byte, ttl time.Duration) error {
+ return c.store(cacheKey, data, nil, ttl, true)
+}
+
+func (c *Cache) store(cacheKey string, byteData []byte, refData cache.ReferenceObject, ttl time.Duration, updateIndex bool) error {
+
+ locks.Acquire(lockPrefix + cacheKey)
+
+ var o1, o2 *index.Object
+ var l int
+ isDirect := byteData == nil && refData != nil
+ if byteData != nil {
+ l = len(byteData)
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "set", "none", float64(l))
+ o1 = &index.Object{Key: cacheKey, Value: byteData, Expiration: time.Now().Add(ttl)}
+ o2 = &index.Object{Key: cacheKey, Value: byteData, Expiration: time.Now().Add(ttl)}
+ } else if refData != nil {
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "setDirect", "none", 0)
+ o1 = &index.Object{Key: cacheKey, ReferenceValue: refData, Expiration: time.Now().Add(ttl)}
+ o2 = &index.Object{Key: cacheKey, ReferenceValue: refData, Expiration: time.Now().Add(ttl)}
+ }
+
+ go log.Debug("memorycache cache store", log.Pairs{"cacheKey": cacheKey, "length": l, "ttl": ttl, "is_direct": isDirect})
+
+ if o1 != nil && o2 != nil {
+ c.client.Store(cacheKey, o1)
+ if updateIndex {
+ c.Index.UpdateObject(o2)
+ }
+ }
+
+ locks.Release(lockPrefix + cacheKey)
+ return nil
+}
+
+// RetrieveReference looks for an object in cache and returns it (or an error if not found)
+func (c *Cache) RetrieveReference(cacheKey string, allowExpired bool) (interface{}, status.LookupStatus, error) {
+ o, s, err := c.retrieve(cacheKey, allowExpired, true)
+ if err != nil {
+ return nil, s, err
+ }
+ if o != nil {
+ return o.ReferenceValue, s, nil
+ }
+ return nil, s, nil
+}
+
+// Retrieve looks for an object in cache and returns it (or an error if not found)
+func (c *Cache) Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error) {
+ o, s, err := c.retrieve(cacheKey, allowExpired, true)
+ if err != nil {
+ return nil, s, err
+ }
+ if o != nil {
+ return o.Value, s, nil
+ }
+ return nil, s, nil
+}
+
+func (c *Cache) retrieve(cacheKey string, allowExpired bool, atime bool) (*index.Object, status.LookupStatus, error) {
+
+ locks.Acquire(lockPrefix + cacheKey)
+
+ record, ok := c.client.Load(cacheKey)
+
+ if ok {
+ o := record.(*index.Object)
+ o.Expiration = c.Index.GetExpiration(cacheKey)
+
+ if allowExpired || o.Expiration.IsZero() || o.Expiration.After(time.Now()) {
+ log.Debug("memory cache retrieve", log.Pairs{"cacheKey": cacheKey})
+ if atime {
+ c.Index.UpdateObjectAccessTime(cacheKey)
+ }
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "get", "hit", float64(len(o.Value)))
+ locks.Release(lockPrefix + cacheKey)
+ return o, status.LookupStatusHit, nil
+ }
+ // Cache Object has been expired but not reaped, go ahead and delete it
+ go c.remove(cacheKey, false)
+ }
+ locks.Release(lockPrefix + cacheKey)
+ _, err := cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ return nil, status.LookupStatusKeyMiss, err
+
+}
+
+// SetTTL updates the TTL for the provided cache object
+func (c *Cache) SetTTL(cacheKey string, ttl time.Duration) {
+ c.Index.UpdateObjectTTL(cacheKey, ttl)
+}
+
+// Remove removes an object from the cache
+func (c *Cache) Remove(cacheKey string) {
+ c.remove(cacheKey, false)
+}
+
+func (c *Cache) remove(cacheKey string, noLock bool) {
+ locks.Acquire(lockPrefix + cacheKey)
+ c.client.Delete(cacheKey)
+ c.Index.RemoveObject(cacheKey, noLock)
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, 0)
+ locks.Release(lockPrefix + cacheKey)
+}
+
+// BulkRemove removes a list of objects from the cache
+func (c *Cache) BulkRemove(cacheKeys []string, noLock bool) {
+ for _, cacheKey := range cacheKeys {
+ c.remove(cacheKey, noLock)
+ }
+}
+
+// Close is not used for Cache, and is here to fully prototype the Cache Interface
+func (c *Cache) Close() error {
+ return nil
+}
diff --git a/internal/cache/memory/memory_test.go b/internal/cache/memory/memory_test.go
new file mode 100644
index 000000000..b32d9f250
--- /dev/null
+++ b/internal/cache/memory/memory_test.go
@@ -0,0 +1,460 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package memory
+
+import (
+ "io/ioutil"
+ "strconv"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/util/log"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+const cacheType = "memory"
+const cacheKey = "cacheKey"
+
+type testReferenceObject struct {
+}
+
+func (r *testReferenceObject) Size() int {
+ return 1
+}
+
+func storeBenchmark(b *testing.B) Cache {
+ log.Logger = log.ConsoleLogger("none")
+ cacheConfig := config.CachingConfig{CacheType: cacheType, Index: config.CacheIndexConfig{ReapInterval: 0}}
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ b.Error(err)
+ }
+ defer mc.Close()
+ for n := 0; n < b.N; n++ {
+ err = mc.Store(cacheKey+strconv.Itoa(n), []byte("data"+strconv.Itoa(n)), time.Duration(60)*time.Second)
+ if err != nil {
+ b.Error(err)
+ }
+ }
+ return mc
+}
+
+func newCacheConfig(t *testing.T) config.CachingConfig {
+ dir, err := ioutil.TempDir("/tmp", cacheType)
+ if err != nil {
+ t.Fatalf("could not create temp directory (%s): %s", dir, err)
+ }
+ return config.CachingConfig{CacheType: cacheType, Index: config.CacheIndexConfig{ReapInterval: 0}}
+}
+
+func TestConfiguration(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+ cfg := mc.Configuration()
+ if cfg.CacheType != cacheType {
+ t.Fatalf("expected %s got %s", cacheType, cfg.CacheType)
+ }
+}
+
+func TestCache_Connect(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ // it should connect
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestCache_StoreReferenceDirect(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ // it should store a value
+ err = mc.store("test", nil, &testReferenceObject{}, 1*time.Second, true)
+
+ r, _, _ := mc.RetrieveReference("test", true)
+ if r == nil {
+ t.Errorf("expected %s got nil", r)
+ }
+
+ _, _, err = mc.RetrieveReference("test2", true)
+ if err == nil {
+ t.Errorf("expected nil got %s", err.Error())
+ }
+
+}
+
+func TestCache_StoreReference(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ // it should store a value
+ err = mc.StoreReference(cacheKey, nil, time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestCache_Store(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ // it should store a value
+ err = mc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkCache_Store(b *testing.B) {
+ storeBenchmark(b)
+}
+
+func TestCache_Retrieve(t *testing.T) {
+
+ const expected1 = `value for key [cacheKey] not in cache`
+
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = mc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ var data []byte
+ var ls status.LookupStatus
+ data, ls, err = mc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\"", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // expire the object
+ mc.SetTTL(cacheKey, -1*time.Hour)
+
+ // this should now return error
+ data, ls, err = mc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ mc.Close()
+ }
+ if err.Error() != expected1 {
+ t.Errorf("expected error '%s' got '%s'", expected1, err.Error())
+ }
+ if string(data) != "" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_Retrieve(b *testing.B) {
+ mc := storeBenchmark(b)
+
+ for n := 0; n < b.N; n++ {
+ var data []byte
+ data, ls, err := mc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\"", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ // expire the object
+ mc.SetTTL(cacheKey+strconv.Itoa(n), -1*time.Hour)
+
+ // this should now return error
+ data, ls, err = mc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ expectederr := `value for key [` + cacheKey + strconv.Itoa(n) + `] not in cache`
+ if err == nil {
+ b.Errorf("expected error for %s", expectederr)
+ mc.Close()
+ }
+ if err.Error() != expectederr {
+ b.Errorf("expected error '%s' got '%s'", expectederr, err.Error())
+ }
+
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestCache_Close(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+ mc.Close()
+}
+
+func TestCache_Remove(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer mc.Close()
+
+ // it should store a value
+ err = mc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := mc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ mc.Remove(cacheKey)
+
+ // it should be a cache miss
+ _, ls, err = mc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_Remove(b *testing.B) {
+ mc := storeBenchmark(b)
+
+ for n := 0; n < b.N; n++ {
+ var data []byte
+ data, ls, err := mc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\"", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ mc.Remove(cacheKey + strconv.Itoa(n))
+
+ // this should now return error
+ data, ls, err = mc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ expectederr := `value for key [` + cacheKey + strconv.Itoa(n) + `] not in cache`
+ if err == nil {
+ b.Errorf("expected error for %s", expectederr)
+ mc.Close()
+ }
+ if err.Error() != expectederr {
+ b.Errorf("expected error '%s' got '%s'", expectederr, err.Error())
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+ if string(data) != "" {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ }
+}
+
+func TestCache_BulkRemove(t *testing.T) {
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer mc.Close()
+
+ // it should store a value
+ err = mc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := mc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ mc.BulkRemove([]string{cacheKey}, true)
+
+ // it should be a cache miss
+ _, ls, err = mc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+
+}
+
+func BenchmarkCache_BulkRemove(b *testing.B) {
+ var keyArray []string
+ for n := 0; n < b.N; n++ {
+ keyArray = append(keyArray, cacheKey+strconv.Itoa(n))
+ }
+
+ mc := storeBenchmark(b)
+
+ mc.BulkRemove(keyArray, true)
+
+ // it should be a cache miss
+ for n := 0; n < b.N; n++ {
+ _, ls, err := mc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestMemoryCache_SetTTL(t *testing.T) {
+
+ cacheConfig := newCacheConfig(t)
+ mc := Cache{Config: &cacheConfig}
+
+ err := mc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer mc.Close()
+
+ exp1 := mc.Index.GetExpiration(cacheKey)
+ if !exp1.IsZero() {
+ t.Errorf("expected Zero time, got %v", exp1)
+ }
+
+ // it should store a value
+ err = mc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ exp1 = mc.Index.GetExpiration(cacheKey)
+ if exp1.IsZero() {
+ t.Errorf("expected time %d, got zero", int(time.Now().Unix())+60)
+ }
+
+ e1 := int(exp1.Unix())
+
+ mc.SetTTL(cacheKey, time.Duration(3600)*time.Second)
+
+ exp2 := mc.Index.GetExpiration(cacheKey)
+ if exp2.IsZero() {
+ t.Errorf("expected time %d, got zero", int(time.Now().Unix())+3600)
+ }
+ e2 := int(exp2.Unix())
+
+ // should be around 3595
+ diff := e2 - e1
+ const expected = 3500
+
+ if diff < expected {
+ t.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, e2, e1)
+ }
+
+}
+
+func BenchmarkCache_SetTTL(b *testing.B) {
+ mc := storeBenchmark(b)
+
+ for n := 0; n < b.N; n++ {
+ exp1 := mc.Index.GetExpiration(cacheKey + strconv.Itoa(n))
+ if exp1.IsZero() {
+ b.Errorf("expected time %d, got zero", int(time.Now().Unix())+60)
+ }
+
+ e1 := int(exp1.Unix())
+
+ mc.SetTTL(cacheKey+strconv.Itoa(n), time.Duration(3600)*time.Second)
+
+ exp2 := mc.Index.GetExpiration(cacheKey + strconv.Itoa(n))
+ if exp2.IsZero() {
+ b.Errorf("expected time %d, got zero", int(time.Now().Unix())+3600)
+ }
+ e2 := int(exp2.Unix())
+
+ // should be around 3595
+ diff := e2 - e1
+ const expected = 3500
+
+ if diff < expected {
+ b.Errorf("expected diff >= %d, got %d from: %d - %d", expected, diff, e2, e1)
+ }
+ }
+}
diff --git a/internal/cache/redis/client_types.go b/internal/cache/redis/client_types.go
new file mode 100644
index 000000000..e219dad00
--- /dev/null
+++ b/internal/cache/redis/client_types.go
@@ -0,0 +1,44 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import "strconv"
+
+// clientType enumerates the methodologies for maintaining time series cache data
+type clientType int
+
+const (
+ clientTypeStandard = clientType(iota)
+ clientTypeCluster
+ clientTypeSentinel
+)
+
+var clientTypeNames = map[string]clientType{
+ "standard": clientTypeStandard,
+ "cluster": clientTypeCluster,
+ "sentinel": clientTypeSentinel,
+}
+
+var clientTypeValues = map[clientType]string{
+ clientTypeStandard: "standard",
+ clientTypeCluster: "cluster",
+ clientTypeSentinel: "sentinel",
+}
+
+func (t clientType) String() string {
+ if v, ok := clientTypeValues[t]; ok {
+ return v
+ }
+ return strconv.Itoa(int(t))
+}
diff --git a/internal/cache/redis/client_types_test.go b/internal/cache/redis/client_types_test.go
new file mode 100644
index 000000000..a4b467ab2
--- /dev/null
+++ b/internal/cache/redis/client_types_test.go
@@ -0,0 +1,33 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import (
+ "testing"
+)
+
+func TestClientTypeString(t *testing.T) {
+
+ t1 := clientTypeStandard
+ var t2 clientType = 20
+
+ if t1.String() != "standard" {
+ t.Errorf("expected %s got %s", "standard", t1.String())
+ }
+
+ if t2.String() != "20" {
+ t.Errorf("expected %s got %s", "20", t2.String())
+ }
+
+}
diff --git a/internal/cache/redis/cluster.go b/internal/cache/redis/cluster.go
new file mode 100644
index 000000000..02ffa08ca
--- /dev/null
+++ b/internal/cache/redis/cluster.go
@@ -0,0 +1,85 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import (
+ "fmt"
+
+ "github.com/go-redis/redis"
+)
+
+func (c *Cache) clusterOpts() (*redis.ClusterOptions, error) {
+
+ if len(c.Config.Redis.Endpoints) == 0 {
+ return nil, fmt.Errorf("Invalid 'endpoints' config")
+ }
+
+ o := &redis.ClusterOptions{
+ Addrs: c.Config.Redis.Endpoints,
+ }
+
+ if c.Config.Redis.Password != "" {
+ o.Password = c.Config.Redis.Password
+ }
+
+ if c.Config.Redis.MaxRetries != 0 {
+ o.MaxRetries = c.Config.Redis.MaxRetries
+ }
+
+ if c.Config.Redis.MinRetryBackoffMS != 0 {
+ o.MinRetryBackoff = durationFromMS(c.Config.Redis.MinRetryBackoffMS)
+ }
+
+ if c.Config.Redis.MaxRetryBackoffMS != 0 {
+ o.MaxRetryBackoff = durationFromMS(c.Config.Redis.MaxRetryBackoffMS)
+ }
+
+ if c.Config.Redis.DialTimeoutMS != 0 {
+ o.DialTimeout = durationFromMS(c.Config.Redis.DialTimeoutMS)
+ }
+
+ if c.Config.Redis.ReadTimeoutMS != 0 {
+ o.ReadTimeout = durationFromMS(c.Config.Redis.ReadTimeoutMS)
+ }
+
+ if c.Config.Redis.WriteTimeoutMS != 0 {
+ o.WriteTimeout = durationFromMS(c.Config.Redis.WriteTimeoutMS)
+ }
+
+ if c.Config.Redis.PoolSize != 0 {
+ o.PoolSize = c.Config.Redis.PoolSize
+ }
+
+ if c.Config.Redis.MinIdleConns != 0 {
+ o.MinIdleConns = c.Config.Redis.MinIdleConns
+ }
+
+ if c.Config.Redis.MaxConnAgeMS != 0 {
+ o.MaxConnAge = durationFromMS(c.Config.Redis.MaxConnAgeMS)
+ }
+
+ if c.Config.Redis.PoolTimeoutMS != 0 {
+ o.PoolTimeout = durationFromMS(c.Config.Redis.PoolTimeoutMS)
+ }
+
+ if c.Config.Redis.IdleTimeoutMS != 0 {
+ o.IdleTimeout = durationFromMS(c.Config.Redis.IdleTimeoutMS)
+ }
+
+ if c.Config.Redis.IdleCheckFrequencyMS != 0 {
+ o.IdleCheckFrequency = durationFromMS(c.Config.Redis.IdleCheckFrequencyMS)
+ }
+
+ return o, nil
+}
diff --git a/internal/cache/redis/redis.go b/internal/cache/redis/redis.go
new file mode 100644
index 000000000..d1ae8a2dc
--- /dev/null
+++ b/internal/cache/redis/redis.go
@@ -0,0 +1,134 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import (
+ "time"
+
+ "github.com/go-redis/redis"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+)
+
+// Redis is the string "redis"
+const Redis = "redis"
+
+// Cache represents a redis cache object that conforms to the Cache interface
+type Cache struct {
+ Name string
+ Config *config.CachingConfig
+
+ client redis.Cmdable
+ closer func() error
+}
+
+// Configuration returns the Configuration for the Cache object
+func (c *Cache) Configuration() *config.CachingConfig {
+ return c.Config
+}
+
+// Connect connects to the configured Redis endpoint
+func (c *Cache) Connect() error {
+ log.Info("connecting to redis", log.Pairs{"protocol": c.Config.Redis.Protocol, "Endpoint": c.Config.Redis.Endpoint})
+
+ switch c.Config.Redis.ClientType {
+ case "sentinel":
+ opts, err := c.sentinelOpts()
+ if err != nil {
+ return err
+ }
+ client := redis.NewFailoverClient(opts)
+ c.closer = client.Close
+ c.client = client
+ case "cluster":
+ opts, err := c.clusterOpts()
+ if err != nil {
+ return err
+ }
+ client := redis.NewClusterClient(opts)
+ c.closer = client.Close
+ c.client = client
+ default:
+ opts, err := c.clientOpts()
+ if err != nil {
+ return err
+ }
+ client := redis.NewClient(opts)
+ c.closer = client.Close
+ c.client = client
+ }
+ return c.client.Ping().Err()
+}
+
+// Store places the the data into the Redis Cache using the provided Key and TTL
+func (c *Cache) Store(cacheKey string, data []byte, ttl time.Duration) error {
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "set", "none", float64(len(data)))
+ log.Debug("redis cache store", log.Pairs{"key": cacheKey})
+ return c.client.Set(cacheKey, data, ttl).Err()
+}
+
+// Retrieve gets data from the Redis Cache using the provided Key
+// because Redis manages Object Expiration internally, allowExpired is not used.
+func (c *Cache) Retrieve(cacheKey string, allowExpired bool) ([]byte, status.LookupStatus, error) {
+ res, err := c.client.Get(cacheKey).Result()
+
+ if err == nil {
+ data := []byte(res)
+ log.Debug("redis cache retrieve", log.Pairs{"key": cacheKey})
+ cache.ObserveCacheOperation(c.Name, c.Config.CacheType, "get", "hit", float64(len(data)))
+ return data, status.LookupStatusHit, nil
+ }
+
+ if err == redis.Nil {
+ log.Debug("redis cache miss", log.Pairs{"key": cacheKey})
+ cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ return nil, status.LookupStatusKeyMiss, err
+ }
+
+ log.Debug("redis cache retrieve failed", log.Pairs{"key": cacheKey, "reason": err.Error()})
+ cache.ObserveCacheMiss(cacheKey, c.Name, c.Config.CacheType)
+ return nil, status.LookupStatusError, err
+}
+
+// Remove removes an object in cache, if present
+func (c *Cache) Remove(cacheKey string) {
+ log.Debug("redis cache remove", log.Pairs{"key": cacheKey})
+ c.client.Del(cacheKey)
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, 0)
+}
+
+// SetTTL updates the TTL for the provided cache object
+func (c *Cache) SetTTL(cacheKey string, ttl time.Duration) {
+ c.client.Expire(cacheKey, ttl)
+}
+
+// BulkRemove removes a list of objects from the cache. noLock is not used for Redis
+func (c *Cache) BulkRemove(cacheKeys []string, noLock bool) {
+ log.Debug("redis cache bulk remove", log.Pairs{})
+ c.client.Del(cacheKeys...)
+ cache.ObserveCacheDel(c.Name, c.Config.CacheType, float64(len(cacheKeys)))
+}
+
+// Close disconnects from the Redis Cache
+func (c *Cache) Close() error {
+ log.Info("closing redis connection", log.Pairs{})
+ return c.closer()
+}
+
+func durationFromMS(input int) time.Duration {
+ return time.Duration(int64(input)) * time.Millisecond
+}
diff --git a/internal/cache/redis/redis_test.go b/internal/cache/redis/redis_test.go
new file mode 100644
index 000000000..66632993e
--- /dev/null
+++ b/internal/cache/redis/redis_test.go
@@ -0,0 +1,519 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import (
+ "strconv"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/util/log"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+
+ "github.com/alicebob/miniredis"
+)
+
+func init() {
+ metrics.Init()
+}
+
+const cacheKey = `cacheKey`
+
+func storeBenchmark(b *testing.B) (*Cache, func()) {
+ log.Logger = log.ConsoleLogger("none")
+ rc, close := setupRedisCache(clientTypeStandard)
+ err := rc.Connect()
+ if err != nil {
+ b.Error(err)
+ }
+ for n := 0; n < b.N; n++ {
+ err := rc.Store(cacheKey+strconv.Itoa(n), []byte("data"+strconv.Itoa(n)), time.Duration(60)*time.Second)
+ if err != nil {
+ b.Error(err)
+ }
+ }
+ return rc, close
+}
+
+func setupRedisCache(ct clientType) (*Cache, func()) {
+ s, err := miniredis.Run()
+ if err != nil {
+ panic(err)
+ }
+ config.Config = config.NewConfig()
+ rcfg := config.RedisCacheConfig{Endpoint: s.Addr(), ClientType: ct.String()}
+ if ct != clientTypeStandard {
+ rcfg.Endpoint = ""
+ rcfg.Endpoints = []string{s.Addr()}
+ if ct == clientTypeSentinel {
+ rcfg.SentinelMaster = s.Addr()
+ }
+ }
+ close := func() {
+ s.Close()
+ }
+ cacheConfig := &config.CachingConfig{CacheType: "redis", Redis: rcfg}
+ config.Caches = map[string]*config.CachingConfig{"default": cacheConfig}
+
+ return &Cache{Config: cacheConfig}, close
+}
+
+func TestClientSelectionSentinel(t *testing.T) {
+ const expected1 = "ERR unknown command `sentinel`"
+ args := []string{"-config", "../../../testdata/test.redis-sentinel.conf",
+ "-origin-url", "http://0.0.0.0", "-origin-type", "rpc", "-log-level", "info"}
+ err := config.Load("trickster", "test", args)
+ if err != nil {
+ t.Error(err)
+ }
+ const cacheName = "test"
+ cfg, ok := config.Caches[cacheName]
+ if !ok {
+ t.Errorf("expected cache named %s", cacheName)
+ }
+ cache := Cache{Name: cacheName, Config: cfg}
+ if err != nil {
+ t.Error(err)
+ }
+ err = cache.Connect()
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ }
+}
+
+func TestSentinelOpts(t *testing.T) {
+
+ const expected1 = `Invalid 'endpoints' config`
+ const expected2 = `Invalid 'sentinel_master' config`
+
+ rc, close := setupRedisCache(clientTypeSentinel)
+ defer close()
+
+ // test empty endpoint
+ rc.Configuration().Redis.Endpoints = nil
+ err := rc.Connect()
+ if err == nil || err.Error() != expected1 {
+ t.Errorf("expected error for %s", expected1)
+ }
+
+ rc.Configuration().Redis.Endpoints = []string{"test"}
+ rc.Configuration().Redis.SentinelMaster = ""
+
+ // test empty SentinelMaster
+ err = rc.Connect()
+ if err == nil || err.Error() != expected2 {
+ t.Errorf("expected error for %s", expected2)
+ }
+}
+
+func TestClusterOpts(t *testing.T) {
+
+ const expected1 = `Invalid 'endpoints' config`
+
+ rc, close := setupRedisCache(clientTypeCluster)
+ defer close()
+
+ // test empty endpoint
+ rc.Configuration().Redis.Endpoints = nil
+ err := rc.Connect()
+ if err == nil || err.Error() != expected1 {
+ t.Errorf("expected error for %s", expected1)
+ }
+}
+
+func TestClientOpts(t *testing.T) {
+
+ const expected1 = `invalid endpoint: `
+
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ // test empty endpoint
+ rc.Configuration().Redis.Endpoint = ""
+ err := rc.Connect()
+ if err == nil || err.Error() != expected1 {
+ t.Errorf("expected error for %s", expected1)
+ }
+}
+
+func TestClientSelectionCluster(t *testing.T) {
+ expected1 := "invalid endpoint"
+ args := []string{"-config", "../../../testdata/test.redis-cluster.conf",
+ "-origin-url", "http://0.0.0.0", "-origin-type", "rpc", "-log-level", "info"}
+ err := config.Load("trickster", "test", args)
+ if err != nil {
+ t.Error(err)
+ }
+ const cacheName = "test"
+ cfg, ok := config.Caches[cacheName]
+ if !ok {
+ t.Errorf("expected cache named %s", cacheName)
+ }
+ cache := Cache{Name: cacheName, Config: cfg}
+ if err != nil {
+ t.Error(err)
+ }
+ err = cache.Connect()
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ }
+}
+
+func TestClientSelectionStandard(t *testing.T) {
+ expected1 := "invalid endpoint"
+ args := []string{"-config", "../../../testdata/test.redis-standard.conf",
+ "-origin-url", "http://0.0.0.0", "-origin-type", "rpc", "-log-level", "info"}
+ err := config.Load("trickster", "test", args)
+ if err != nil {
+ t.Error(err)
+ }
+ const cacheName = "test"
+ cfg, ok := config.Caches[cacheName]
+ if !ok {
+ t.Errorf("expected cache named %s", cacheName)
+ }
+ cache := Cache{Name: cacheName, Config: cfg}
+ if err != nil {
+ t.Error(err)
+ }
+ err = cache.Connect()
+ if err == nil {
+ t.Errorf("expected error for %s", expected1)
+ }
+}
+
+func TestDurationFromMS(t *testing.T) {
+
+ tests := []struct {
+ input int
+ expected time.Duration
+ }{
+ {0, time.Duration(0)},
+ {5000, time.Duration(5000) * time.Millisecond},
+ {60000, time.Duration(60000) * time.Millisecond},
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+
+ res := durationFromMS(test.input)
+
+ if res != test.expected {
+ t.Fatalf("Mismatch in durationFromMS: expected=%f actual=%f", test.expected.Seconds(), res.Seconds())
+ }
+ })
+ }
+
+}
+
+func TestRedisCache_SetTTL(t *testing.T) {
+
+ const expected = "data"
+
+ cache, closer := setupRedisCache(clientTypeStandard)
+ defer closer()
+
+ err := cache.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer cache.Close()
+
+ // it should store a value
+ err = cache.Store(cacheKey, []byte(expected), time.Duration(1)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+ cache.SetTTL(cacheKey, time.Duration(3600)*time.Second)
+
+ // since the TTL is updated to 1 hour, waiting more than the original TTL of 1s
+ // should not matter
+ time.Sleep(1010 * time.Millisecond)
+
+ val, ls, err := cache.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ if string(val) != expected {
+ t.Errorf("expected %s got %s", expected, string(val))
+ }
+
+}
+
+func BenchmarkCache_SetTTL(b *testing.B) {
+ rc, close := storeBenchmark(b)
+ defer close()
+
+ for n := 0; n < b.N; n++ {
+ expected := "data" + strconv.Itoa(n)
+ rc.SetTTL(cacheKey+strconv.Itoa(n), time.Duration(3600)*time.Second)
+ //time.Sleep(1010 * time.Millisecond)
+ val, ls, err := rc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(val) != expected {
+ b.Errorf("expected %s got %s", expected, string(val))
+ }
+ }
+}
+
+func TestConfiguration(t *testing.T) {
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ cfg := rc.Configuration()
+ if cfg.Redis.ClientType != clientTypeStandard.String() {
+ t.Fatalf("expected %s got %s", clientTypeStandard.String(), cfg.Redis.ClientType)
+ }
+}
+
+func TestRedisCache_Connect(t *testing.T) {
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ // it should connect
+ err := rc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestRedisCache_Store(t *testing.T) {
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ err := rc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should store a value
+ err = rc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkCache_Store(b *testing.B) {
+ rc, close := storeBenchmark(b)
+ if rc == nil {
+ b.Error("Could not create the redis cache")
+ }
+ defer close()
+}
+
+func TestRedisCache_Retrieve(t *testing.T) {
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ err := rc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ err = rc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := rc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\"", "data", data)
+ }
+}
+
+func BenchmarkCache_Retrieve(b *testing.B) {
+ rc, close := storeBenchmark(b)
+ defer close()
+
+ for n := 0; n < b.N; n++ {
+ data, ls, err := rc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\".", "data"+strconv.Itoa(n), data)
+ }
+ }
+}
+
+func TestRedisCache_Close(t *testing.T) {
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ err := rc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should close
+ err = rc.Close()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestCache_Remove(t *testing.T) {
+
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ err := rc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer rc.Close()
+
+ // it should store a value
+ err = rc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := rc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ rc.Remove(cacheKey)
+
+ // it should be a cache miss
+ _, ls, err = rc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_Remove(b *testing.B) {
+ rc, close := storeBenchmark(b)
+ defer close()
+
+ for n := 0; n < b.N; n++ {
+ var data []byte
+ data, ls, err := rc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err != nil {
+ b.Error(err)
+ }
+ if string(data) != "data"+strconv.Itoa(n) {
+ b.Errorf("wanted \"%s\". got \"%s\"", "data"+strconv.Itoa(n), data)
+ }
+ if ls != status.LookupStatusHit {
+ b.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ rc.Remove(cacheKey + strconv.Itoa(n))
+
+ _, ls, err = rc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected key not found error for %s", cacheKey+strconv.Itoa(n))
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
+
+func TestCache_BulkRemove(t *testing.T) {
+
+ rc, close := setupRedisCache(clientTypeStandard)
+ defer close()
+
+ err := rc.Connect()
+ if err != nil {
+ t.Error(err)
+ }
+ defer rc.Close()
+
+ // it should store a value
+ err = rc.Store(cacheKey, []byte("data"), time.Duration(60)*time.Second)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // it should retrieve a value
+ data, ls, err := rc.Retrieve(cacheKey, false)
+ if err != nil {
+ t.Error(err)
+ }
+ if string(data) != "data" {
+ t.Errorf("wanted \"%s\". got \"%s\".", "data", data)
+ }
+ if ls != status.LookupStatusHit {
+ t.Errorf("expected %s got %s", status.LookupStatusHit, ls)
+ }
+
+ rc.BulkRemove([]string{cacheKey}, true)
+
+ // it should be a cache miss
+ _, ls, err = rc.Retrieve(cacheKey, false)
+ if err == nil {
+ t.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+}
+
+func BenchmarkCache_BulkRemove(b *testing.B) {
+ rc, close := storeBenchmark(b)
+ defer close()
+
+ var keyArray []string
+ for n := 0; n < b.N; n++ {
+ keyArray = append(keyArray, cacheKey+strconv.Itoa(n))
+ }
+
+ rc.BulkRemove(keyArray, true)
+
+ // it should be a cache miss
+ for n := 0; n < b.N; n++ {
+ _, ls, err := rc.Retrieve(cacheKey+strconv.Itoa(n), false)
+ if err == nil {
+ b.Errorf("expected key not found error for %s", cacheKey)
+ }
+ if ls != status.LookupStatusKeyMiss {
+ b.Errorf("expected %s got %s", status.LookupStatusKeyMiss, ls)
+ }
+ }
+}
diff --git a/internal/cache/redis/sentinel.go b/internal/cache/redis/sentinel.go
new file mode 100644
index 000000000..794d0cffc
--- /dev/null
+++ b/internal/cache/redis/sentinel.go
@@ -0,0 +1,94 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import (
+ "fmt"
+
+ "github.com/go-redis/redis"
+)
+
+func (c *Cache) sentinelOpts() (*redis.FailoverOptions, error) {
+
+ if len(c.Config.Redis.Endpoints) == 0 {
+ return nil, fmt.Errorf("Invalid 'endpoints' config")
+ }
+
+ if c.Config.Redis.SentinelMaster == "" {
+ return nil, fmt.Errorf("Invalid 'sentinel_master' config")
+ }
+
+ o := &redis.FailoverOptions{
+ SentinelAddrs: c.Config.Redis.Endpoints,
+ MasterName: c.Config.Redis.SentinelMaster,
+ }
+
+ if c.Config.Redis.Password != "" {
+ o.Password = c.Config.Redis.Password
+ }
+
+ if c.Config.Redis.DB != 0 {
+ o.DB = c.Config.Redis.DB
+ }
+
+ if c.Config.Redis.MaxRetries != 0 {
+ o.MaxRetries = c.Config.Redis.MaxRetries
+ }
+
+ if c.Config.Redis.MinRetryBackoffMS != 0 {
+ o.MinRetryBackoff = durationFromMS(c.Config.Redis.MinRetryBackoffMS)
+ }
+
+ if c.Config.Redis.MaxRetryBackoffMS != 0 {
+ o.MaxRetryBackoff = durationFromMS(c.Config.Redis.MaxRetryBackoffMS)
+ }
+
+ if c.Config.Redis.DialTimeoutMS != 0 {
+ o.DialTimeout = durationFromMS(c.Config.Redis.DialTimeoutMS)
+ }
+
+ if c.Config.Redis.ReadTimeoutMS != 0 {
+ o.ReadTimeout = durationFromMS(c.Config.Redis.ReadTimeoutMS)
+ }
+
+ if c.Config.Redis.WriteTimeoutMS != 0 {
+ o.WriteTimeout = durationFromMS(c.Config.Redis.WriteTimeoutMS)
+ }
+
+ if c.Config.Redis.PoolSize != 0 {
+ o.PoolSize = c.Config.Redis.PoolSize
+ }
+
+ if c.Config.Redis.MinIdleConns != 0 {
+ o.MinIdleConns = c.Config.Redis.MinIdleConns
+ }
+
+ if c.Config.Redis.MaxConnAgeMS != 0 {
+ o.MaxConnAge = durationFromMS(c.Config.Redis.MaxConnAgeMS)
+ }
+
+ if c.Config.Redis.PoolTimeoutMS != 0 {
+ o.PoolTimeout = durationFromMS(c.Config.Redis.PoolTimeoutMS)
+ }
+
+ if c.Config.Redis.IdleTimeoutMS != 0 {
+ o.IdleTimeout = durationFromMS(c.Config.Redis.IdleTimeoutMS)
+ }
+
+ if c.Config.Redis.IdleCheckFrequencyMS != 0 {
+ o.IdleCheckFrequency = durationFromMS(c.Config.Redis.IdleCheckFrequencyMS)
+ }
+
+ return o, nil
+}
diff --git a/internal/cache/redis/standard.go b/internal/cache/redis/standard.go
new file mode 100644
index 000000000..cfe2a1ac8
--- /dev/null
+++ b/internal/cache/redis/standard.go
@@ -0,0 +1,93 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package redis
+
+import (
+ "fmt"
+
+ "github.com/go-redis/redis"
+)
+
+func (c *Cache) clientOpts() (*redis.Options, error) {
+
+ if c.Config.Redis.Endpoint == "" {
+ return nil, fmt.Errorf("invalid endpoint: %s", c.Config.Redis.Endpoint)
+ }
+
+ o := &redis.Options{
+ Addr: c.Config.Redis.Endpoint,
+ }
+
+ if c.Config.Redis.Protocol != "" {
+ o.Network = c.Config.Redis.Protocol
+ }
+
+ if c.Config.Redis.Password != "" {
+ o.Password = c.Config.Redis.Password
+ }
+
+ if c.Config.Redis.DB != 0 {
+ o.DB = c.Config.Redis.DB
+ }
+
+ if c.Config.Redis.MaxRetries != 0 {
+ o.MaxRetries = c.Config.Redis.MaxRetries
+ }
+
+ if c.Config.Redis.MinRetryBackoffMS != 0 {
+ o.MinRetryBackoff = durationFromMS(c.Config.Redis.MinRetryBackoffMS)
+ }
+
+ if c.Config.Redis.MaxRetryBackoffMS != 0 {
+ o.MaxRetryBackoff = durationFromMS(c.Config.Redis.MaxRetryBackoffMS)
+ }
+
+ if c.Config.Redis.DialTimeoutMS != 0 {
+ o.DialTimeout = durationFromMS(c.Config.Redis.DialTimeoutMS)
+ }
+
+ if c.Config.Redis.ReadTimeoutMS != 0 {
+ o.ReadTimeout = durationFromMS(c.Config.Redis.ReadTimeoutMS)
+ }
+
+ if c.Config.Redis.WriteTimeoutMS != 0 {
+ o.WriteTimeout = durationFromMS(c.Config.Redis.WriteTimeoutMS)
+ }
+
+ if c.Config.Redis.PoolSize != 0 {
+ o.PoolSize = c.Config.Redis.PoolSize
+ }
+
+ if c.Config.Redis.MinIdleConns != 0 {
+ o.MinIdleConns = c.Config.Redis.MinIdleConns
+ }
+
+ if c.Config.Redis.MaxConnAgeMS != 0 {
+ o.MaxConnAge = durationFromMS(c.Config.Redis.MaxConnAgeMS)
+ }
+
+ if c.Config.Redis.PoolTimeoutMS != 0 {
+ o.PoolTimeout = durationFromMS(c.Config.Redis.PoolTimeoutMS)
+ }
+
+ if c.Config.Redis.IdleTimeoutMS != 0 {
+ o.IdleTimeout = durationFromMS(c.Config.Redis.IdleTimeoutMS)
+ }
+
+ if c.Config.Redis.IdleCheckFrequencyMS != 0 {
+ o.IdleCheckFrequency = durationFromMS(c.Config.Redis.IdleCheckFrequencyMS)
+ }
+
+ return o, nil
+}
diff --git a/internal/cache/registration/registration.go b/internal/cache/registration/registration.go
new file mode 100644
index 000000000..afba3e826
--- /dev/null
+++ b/internal/cache/registration/registration.go
@@ -0,0 +1,76 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package registration
+
+import (
+ "fmt"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/badger"
+ "github.com/Comcast/trickster/internal/cache/bbolt"
+ "github.com/Comcast/trickster/internal/cache/filesystem"
+ "github.com/Comcast/trickster/internal/cache/memory"
+ "github.com/Comcast/trickster/internal/cache/redis"
+ "github.com/Comcast/trickster/internal/config"
+)
+
+// Cache Interface Types
+const (
+ ctFilesystem = "filesystem"
+ ctRedis = "redis"
+ ctBBolt = "bbolt"
+ ctBadger = "badger"
+)
+
+// Caches maintains a list of active caches
+var Caches = make(map[string]cache.Cache)
+
+// GetCache returns the Cache named cacheName if it exists
+func GetCache(cacheName string) (cache.Cache, error) {
+ if c, ok := Caches[cacheName]; ok {
+ return c, nil
+ }
+ return nil, fmt.Errorf("Could not find Cache named [%s]", cacheName)
+}
+
+// LoadCachesFromConfig iterates the Caching Confi and Connects/Maps each Cache
+func LoadCachesFromConfig() {
+ for k, v := range config.Caches {
+ c := NewCache(k, v)
+ Caches[k] = c
+ }
+}
+
+// NewCache returns a Cache object based on the provided config.CachingConfig
+func NewCache(cacheName string, cfg *config.CachingConfig) cache.Cache {
+
+ var c cache.Cache
+
+ switch cfg.CacheType {
+ case ctFilesystem:
+ c = &filesystem.Cache{Name: cacheName, Config: cfg}
+ case ctRedis:
+ c = &redis.Cache{Name: cacheName, Config: cfg}
+ case ctBBolt:
+ c = &bbolt.Cache{Name: cacheName, Config: cfg}
+ case ctBadger:
+ c = &badger.Cache{Name: cacheName, Config: cfg}
+ default:
+ // Default to MemoryCache
+ c = &memory.Cache{Name: cacheName, Config: cfg}
+ }
+
+ c.Connect()
+ return c
+}
diff --git a/internal/cache/registration/registration_test.go b/internal/cache/registration/registration_test.go
new file mode 100644
index 000000000..5f7ce6625
--- /dev/null
+++ b/internal/cache/registration/registration_test.go
@@ -0,0 +1,109 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package registration
+
+import (
+ "io/ioutil"
+ "os"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+func TestLoadCachesFromConfig(t *testing.T) {
+
+ err := config.Load("trickster", "test", []string{"-log-level", "debug", "-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ for key, v := range config.CacheTypeNames {
+ cfg := newCacheConfig(t, key)
+ config.Caches[key] = cfg
+ switch v {
+ case config.CacheTypeBbolt:
+ defer os.RemoveAll(cfg.BBolt.Filename)
+ case config.CacheTypeFilesystem:
+ defer os.RemoveAll(cfg.Filesystem.CachePath)
+ case config.CacheTypeBadgerDB:
+ defer os.RemoveAll(cfg.Badger.Directory)
+ }
+ }
+
+ LoadCachesFromConfig()
+ _, err = GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ for key := range config.CacheTypeNames {
+ _, err = GetCache(key)
+ if err != nil {
+ t.Error(err)
+ }
+ }
+
+ _, err = GetCache("foo")
+ if err == nil {
+ t.Errorf("expected error")
+ }
+
+}
+
+func newCacheConfig(t *testing.T, cacheType string) *config.CachingConfig {
+
+ bd := "."
+ fd := "."
+ var err error
+
+ ctid, ok := config.CacheTypeNames[cacheType]
+ if !ok {
+ ctid = config.CacheTypeMemory
+ }
+
+ switch ctid {
+ case config.CacheTypeBadgerDB:
+ bd, err = ioutil.TempDir("/tmp", cacheType)
+ if err != nil {
+ t.Error(err)
+ }
+
+ case config.CacheTypeFilesystem:
+ fd, err = ioutil.TempDir("/tmp", cacheType)
+ if err != nil {
+ t.Error(err)
+ }
+ }
+
+ return &config.CachingConfig{
+ CacheType: cacheType,
+ Redis: config.RedisCacheConfig{Protocol: "tcp", Endpoint: "redis:6379", Endpoints: []string{"redis:6379"}},
+ Filesystem: config.FilesystemCacheConfig{CachePath: fd},
+ BBolt: config.BBoltCacheConfig{Filename: "/tmp/test.db", Bucket: "trickster_test"},
+ Badger: config.BadgerCacheConfig{Directory: bd, ValueDirectory: bd},
+ Index: config.CacheIndexConfig{
+ ReapIntervalSecs: 3,
+ FlushIntervalSecs: 5,
+ MaxSizeBytes: 536870912,
+ MaxSizeBackoffBytes: 16777216,
+ MaxSizeObjects: 0,
+ MaxSizeBackoffObjects: 100,
+ },
+ }
+}
diff --git a/internal/cache/status/lookup_status.go b/internal/cache/status/lookup_status.go
new file mode 100644
index 000000000..b84bf40ca
--- /dev/null
+++ b/internal/cache/status/lookup_status.go
@@ -0,0 +1,79 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package status
+
+import "strconv"
+
+// LookupStatus defines the possible status of a cache lookup
+type LookupStatus int
+
+const (
+ // LookupStatusHit indicates a full cache hit on lookup
+ LookupStatusHit = LookupStatus(iota)
+ // LookupStatusPartialHit indicates a partial cache hit (key exists and has some data
+ // for requested time range, but not all) on lookup
+ LookupStatusPartialHit
+ // LookupStatusRangeMiss indicates a range miss (key exists but no data for requested time range) on lookup
+ LookupStatusRangeMiss
+ // LookupStatusKeyMiss indicates a full key miss (cache key does not exist) on lookup
+ LookupStatusKeyMiss
+ // LookupStatusRevalidated indicates the cached object execeeded the freshness lifetime but
+ // was revalidated against the upstream server and is treated as a cache hit
+ LookupStatusRevalidated
+ // LookupStatusPurge indicates the cache key, if it existed, was purged as directed
+ // in upstream response or down stream request http headers
+ LookupStatusPurge
+ // LookupStatusProxyError indicates that a proxy error occurred retrieving a cacheable dataset
+ // in upstream response or down stream request http headers
+ LookupStatusProxyError
+ // LookupStatusProxyOnly indicates that the request was fully proxied to the origin without using the cache
+ LookupStatusProxyOnly
+ // LookupStatusNegativeCacheHit indicates that the request was served as a hit from the Negative Response Cache
+ LookupStatusNegativeCacheHit
+ // LookupStatusError indicates that there was an error looking up the object in the cache
+ LookupStatusError
+)
+
+var cacheLookupStatusNames = map[string]LookupStatus{
+ "hit": LookupStatusHit,
+ "phit": LookupStatusPartialHit,
+ "rhit": LookupStatusRevalidated,
+ "rmiss": LookupStatusRangeMiss,
+ "kmiss": LookupStatusKeyMiss,
+ "purge": LookupStatusPurge,
+ "proxy-error": LookupStatusProxyError,
+ "proxy-only": LookupStatusProxyOnly,
+ "nchit": LookupStatusNegativeCacheHit,
+ "error": LookupStatusError,
+}
+
+var cacheLookupStatusValues = map[LookupStatus]string{
+ LookupStatusHit: "hit",
+ LookupStatusPartialHit: "phit",
+ LookupStatusRevalidated: "rhit",
+ LookupStatusRangeMiss: "rmiss",
+ LookupStatusKeyMiss: "kmiss",
+ LookupStatusPurge: "purge",
+ LookupStatusProxyError: "proxy-error",
+ LookupStatusProxyOnly: "proxy-only",
+ LookupStatusNegativeCacheHit: "nchit",
+ LookupStatusError: "error",
+}
+
+func (s LookupStatus) String() string {
+ if v, ok := cacheLookupStatusValues[s]; ok {
+ return v
+ }
+ return strconv.Itoa(int(s))
+}
diff --git a/internal/cache/status/lookup_status_test.go b/internal/cache/status/lookup_status_test.go
new file mode 100644
index 000000000..592fdac93
--- /dev/null
+++ b/internal/cache/status/lookup_status_test.go
@@ -0,0 +1,36 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package status
+
+import "testing"
+
+func TestLookupStatusString(t *testing.T) {
+
+ t1 := LookupStatusHit
+ t2 := LookupStatusKeyMiss
+
+ var t3 LookupStatus = 10
+
+ if t1.String() != "hit" {
+ t.Errorf("expected %s got %s", "hit", t1.String())
+ }
+
+ if t2.String() != "kmiss" {
+ t.Errorf("expected %s got %s", "kmiss", t2.String())
+ }
+
+ if t3.String() != "10" {
+ t.Errorf("expected %s got %s", "9", t3.String())
+ }
+}
diff --git a/internal/config/cache_type.go b/internal/config/cache_type.go
new file mode 100644
index 000000000..38ff5860f
--- /dev/null
+++ b/internal/config/cache_type.go
@@ -0,0 +1,57 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import "strconv"
+
+// CacheType enumerates the methodologies for maintaining time series cache data
+type CacheType int
+
+const (
+ // CacheTypeMemory indicates a memory cache
+ CacheTypeMemory = CacheType(iota)
+ // CacheTypeFilesystem indicates a filesystem cache
+ CacheTypeFilesystem
+ // CacheTypeRedis indicates a Redis cache
+ CacheTypeRedis
+ // CacheTypeBbolt indicates a Bbolt cache
+ CacheTypeBbolt
+ // CacheTypeBadgerDB indicates a BadgerDB cache
+ CacheTypeBadgerDB
+)
+
+// CacheTypeNames is a map of cache types keyed by name
+var CacheTypeNames = map[string]CacheType{
+ "memory": CacheTypeMemory,
+ "filesystem": CacheTypeFilesystem,
+ "redis": CacheTypeRedis,
+ "bbolt": CacheTypeBbolt,
+ "badger": CacheTypeBadgerDB,
+}
+
+// CacheTypeValues is a map of cache types keyed by internal id
+var CacheTypeValues = map[CacheType]string{
+ CacheTypeMemory: "memory",
+ CacheTypeFilesystem: "filesystem",
+ CacheTypeRedis: "redis",
+ CacheTypeBbolt: "bbolt",
+ CacheTypeBadgerDB: "badger",
+}
+
+func (t CacheType) String() string {
+ if v, ok := CacheTypeValues[t]; ok {
+ return v
+ }
+ return strconv.Itoa(int(t))
+}
diff --git a/internal/config/cache_type_test.go b/internal/config/cache_type_test.go
new file mode 100644
index 000000000..75f9488ab
--- /dev/null
+++ b/internal/config/cache_type_test.go
@@ -0,0 +1,38 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "testing"
+)
+
+func TestCacheTypeString(t *testing.T) {
+
+ t1 := CacheTypeMemory
+ t2 := CacheTypeFilesystem
+ var t3 CacheType = 13
+
+ if t1.String() != "memory" {
+ t.Errorf("expected %s got %s", "memory", t1.String())
+ }
+
+ if t2.String() != "filesystem" {
+ t.Errorf("expected %s got %s", "filesystem", t2.String())
+ }
+
+ if t3.String() != "13" {
+ t.Errorf("expected %s got %s", "13", t3.String())
+ }
+
+}
diff --git a/internal/config/coll_fwd_type.go b/internal/config/coll_fwd_type.go
new file mode 100644
index 000000000..b9b847575
--- /dev/null
+++ b/internal/config/coll_fwd_type.go
@@ -0,0 +1,45 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import "strconv"
+
+// CollapsedForwardingType enumerates the methodologies for maintaining time series cache data
+type CollapsedForwardingType int
+
+const (
+ // CFTypeBasic indicates a basic cache
+ CFTypeBasic = CollapsedForwardingType(iota)
+ // CFTypeProgressive indicates a progressive cache
+ CFTypeProgressive
+)
+
+// CollapsedForwardingTypeNames is a map of cache types keyed by name
+var CollapsedForwardingTypeNames = map[string]CollapsedForwardingType{
+ "basic": CFTypeBasic,
+ "progressive": CFTypeProgressive,
+}
+
+// CollapsedForwardingTypeValues is a map of cache types keyed by internal id
+var CollapsedForwardingTypeValues = map[CollapsedForwardingType]string{
+ CFTypeBasic: "basic",
+ CFTypeProgressive: "progressive",
+}
+
+func (t CollapsedForwardingType) String() string {
+ if v, ok := CollapsedForwardingTypeValues[t]; ok {
+ return v
+ }
+ return strconv.Itoa(int(t))
+}
diff --git a/internal/config/coll_fwd_type_test.go b/internal/config/coll_fwd_type_test.go
new file mode 100644
index 000000000..dafbad30d
--- /dev/null
+++ b/internal/config/coll_fwd_type_test.go
@@ -0,0 +1,38 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "testing"
+)
+
+func TestCollapsedForwardingTypeString(t *testing.T) {
+
+ t1 := CFTypeBasic
+ t2 := CFTypeProgressive
+ var t3 CollapsedForwardingType = 13
+
+ if t1.String() != "basic" {
+ t.Errorf("expected %s got %s", "basic", t1.String())
+ }
+
+ if t2.String() != "progressive" {
+ t.Errorf("expected %s got %s", "progressive", t2.String())
+ }
+
+ if t3.String() != "13" {
+ t.Errorf("expected %s got %s", "13", t3.String())
+ }
+
+}
diff --git a/internal/config/config.go b/internal/config/config.go
new file mode 100644
index 000000000..9d17a4861
--- /dev/null
+++ b/internal/config/config.go
@@ -0,0 +1,1028 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "bytes"
+ "fmt"
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/BurntSushi/toml"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+)
+
+// Config is the Running Configuration for Trickster
+var Config *TricksterConfig
+
+// Main is the Main subsection of the Running Configuration
+var Main *MainConfig
+
+// Origins is the Origin Map subsection of the Running Configuration
+var Origins map[string]*OriginConfig
+
+// Caches is the Cache Map subsection of the Running Configuration
+var Caches map[string]*CachingConfig
+
+// Frontend is the Proxy Server subsection of the Running Configuration
+var Frontend *FrontendConfig
+
+// Logging is the Logging subsection of the Running Configuration
+var Logging *LoggingConfig
+
+// Metrics is the Metrics subsection of the Running Configuration
+var Metrics *MetricsConfig
+
+// NegativeCacheConfigs is the NegativeCacheConfig subsection of the Running Configuration
+var NegativeCacheConfigs map[string]NegativeCacheConfig
+
+// Flags is a collection of command line flags that Trickster loads.
+var Flags = TricksterFlags{}
+var providedOriginURL string
+var providedOriginType string
+
+// LoaderWarnings holds warnings generated during config load (before the logger is initialized),
+// so they can be logged at the end of the loading process
+var LoaderWarnings = make([]string, 0)
+
+// TricksterConfig is the main configuration object
+type TricksterConfig struct {
+ // Main is the primary MainConfig section
+ Main *MainConfig `toml:"main"`
+ // Origins is a map of OriginConfigs
+ Origins map[string]*OriginConfig `toml:"origins"`
+ // Caches is a map of CacheConfigs
+ Caches map[string]*CachingConfig `toml:"caches"`
+ // ProxyServer is provides configurations about the Proxy Front End
+ Frontend *FrontendConfig `toml:"frontend"`
+ // Logging provides configurations that affect logging behavior
+ Logging *LoggingConfig `toml:"logging"`
+ // Metrics provides configurations for collecting Metrics about the application
+ Metrics *MetricsConfig `toml:"metrics"`
+ // NegativeCacheConfigs is a map of NegativeCacheConfigs
+ NegativeCacheConfigs map[string]NegativeCacheConfig `toml:"negative_caches"`
+
+ activeCaches map[string]bool
+}
+
+// MainConfig is a collection of general configuration values.
+type MainConfig struct {
+ // InstanceID represents a unique ID for the current instance, when multiple instances on the same host
+ InstanceID int `toml:"instance_id"`
+ // ConfigHandlerPath provides the path to register the Config Handler for outputting the running configuration
+ ConfigHandlerPath string `toml:"config_handler_path"`
+ // PingHandlerPath provides the path to register the Ping Handler for checking that Trickster is running
+ PingHandlerPath string `toml:"ping_handler_path"`
+}
+
+// OriginConfig is a collection of configurations for prometheus origins proxied by Trickster
+type OriginConfig struct {
+
+ // HTTP and Proxy Configurations
+ //
+ // IsDefault indicates if this is the default origin for any request not matching a configured route
+ IsDefault bool `toml:"is_default"`
+ // OriginType describes the type of origin (e.g., 'prometheus')
+ OriginType string `toml:"origin_type"`
+ // OriginURL provides the base upstream URL for all proxied requests to this origin.
+ // it can be as simple as http://example.com or as complex as https://example.com:8443/path/prefix
+ OriginURL string `toml:"origin_url"`
+ // TimeoutSecs defines how long the HTTP request will wait for a response before timing out
+ TimeoutSecs int64 `toml:"timeout_secs"`
+ // KeepAliveTimeoutSecs defines how long an open keep-alive HTTP connection remains idle before closing
+ KeepAliveTimeoutSecs int64 `toml:"keep_alive_timeout_secs"`
+ // MaxIdleConns defines maximum number of open keep-alive connections to maintain
+ MaxIdleConns int `toml:"max_idle_conns"`
+ // CacheName provides the name of the configured cache where the origin client will store it's cache data
+ CacheName string `toml:"cache_name"`
+ // CacheKeyPrefix defines the cache key prefix the origin will use when writing objects to the cache
+ CacheKeyPrefix string `toml:"cache_key_prefix"`
+ // HealthCheckUpstreamPath provides the URL path for the upstream health check
+ HealthCheckUpstreamPath string `toml:"health_check_upstream_path"`
+ // HealthCheckVerb provides the HTTP verb to use when making an upstream health check
+ HealthCheckVerb string `toml:"health_check_verb"`
+ // HealthCheckQuery provides the HTTP query parameters to use when making an upstream health check
+ HealthCheckQuery string `toml:"health_check_query"`
+ // HealthCheckHeaders provides the HTTP Headers to apply when making an upstream health check
+ HealthCheckHeaders map[string]string `toml:"health_check_headers"`
+ // Object Proxy Cache and Delta Proxy Cache Configurations
+ // TimeseriesRetentionFactor limits the maximum the number of chronological timestamps worth of data to store in cache for each query
+ TimeseriesRetentionFactor int `toml:"timeseries_retention_factor"`
+ // TimeseriesEvictionMethodName specifies which methodology ("oldest", "lru") is used to identify timeseries to evict from a full cache object
+ TimeseriesEvictionMethodName string `toml:"timeseries_eviction_method"`
+ // FastForwardDisable indicates whether the FastForward feature should be disabled for this origin
+ FastForwardDisable bool `toml:"fast_forward_disable"`
+ // BackfillToleranceSecs prevents values with timestamps newer than the provided number of seconds from being cached
+ // this allows propagation of upstream backfill operations that modify recently-served data
+ BackfillToleranceSecs int64 `toml:"backfill_tolerance_secs"`
+ // PathList is a list of PathConfigs that control the behavior of the given paths when requested
+ Paths map[string]*PathConfig `toml:"paths"`
+ // NegativeCacheName provides the name of the Negative Cache Config to be used by this Origin
+ NegativeCacheName string `toml:"negative_cache_name"`
+ // TimeseriesTTLSecs specifies the cache TTL of timeseries objects
+ TimeseriesTTLSecs int `toml:"timeseries_ttl_secs"`
+ // TimeseriesTTLSecs specifies the cache TTL of fast forward data
+ FastForwardTTLSecs int `toml:"fastforward_ttl_secs"`
+ // MaxTTLSecs specifies the maximum allowed TTL for any cache object
+ MaxTTLSecs int `toml:"max_ttl_secs"`
+ // RevalidationFactor specifies how many times to multiply the object freshness lifetime by to calculate an absolute cache TTL
+ RevalidationFactor float64 `toml:"revalidation_factor"`
+ // MaxObjectSizeBytes specifies the max objectsize to be accepted for any given cache object
+ MaxObjectSizeBytes int `toml:"max_object_size_bytes"`
+ // CompressableTypeList specifies the HTTP Object Content Types that will be compressed internally when stored in the Trickster cache
+ CompressableTypeList []string `toml:"compressable_types"`
+
+ // TLS is the TLS Configuration for the Frontend and Backend
+ TLS *TLSConfig `toml:"tls"`
+ // RequireTLS, when true, indicates this Origin Config's paths must only be registered with the TLS Router
+ RequireTLS bool `toml:"require_tls"`
+
+ // MultipartRangesDisabled, when true, indicates that if a downstream client requests multiple ranges in a single Range request,
+ // Trickster will instead request and return a 200 OK with the full object body
+ MultipartRangesDisabled bool `toml:"multipart_ranges_disabled"`
+ // DearticulateUpstreamRanges, when true, indicates that when Trickster requests multiple ranges from the origin,
+ // that they be requested as individual upstream requests instead of a single request that expects a multipart response
+ // this optimizes Trickster to request as few bytes as possible when fronting origins that only support single range requests
+ DearticulateUpstreamRanges bool `toml:"dearticulate_upstream_ranges"`
+
+ // Synthesized Configurations
+ // These configurations are parsed versions of those defined above, and are what Trickster uses internally
+ //
+ // Name is the Name of the origin, taken from the Key in the Origins map[string]*OriginConfig
+ Name string `toml:"-"`
+ // Timeout is the time.Duration representation of TimeoutSecs
+ Timeout time.Duration `toml:"-"`
+ // BackfillTolerance is the time.Duration representation of BackfillToleranceSecs
+ BackfillTolerance time.Duration `toml:"-"`
+ // ValueRetention is the time.Duration representation of ValueRetentionSecs
+ ValueRetention time.Duration `toml:"-"`
+ // Scheme is the layer 7 protocol indicator (e.g. 'http'), derived from OriginURL
+ Scheme string `toml:"-"`
+ // Host is the upstream hostname/IP[:port] the origin client will connect to when fetching uncached data, derived from OriginURL
+ Host string `toml:"-"`
+ // PathPrefix provides any prefix added to the front of the requested path when constructing the upstream request url, derived from OriginURL
+ PathPrefix string `toml:"-"`
+ // NegativeCache provides a map for the negative cache, with TTLs converted to time.Durations
+ NegativeCache map[int]time.Duration `toml:"-"`
+ // TimeseriesRetention when subtracted from time.Now() represents the oldest allowable timestamp in a timeseries when EvictionMethod is 'oldest'
+ TimeseriesRetention time.Duration `toml:"-"`
+ // TimeseriesEvictionMethod is the parsed value of TimeseriesEvictionMethodName
+ TimeseriesEvictionMethod TimeseriesEvictionMethod `toml:"-"`
+ // TimeseriesTTL is the parsed value of TimeseriesTTLSecs
+ TimeseriesTTL time.Duration `toml:"-"`
+ // FastForwardTTL is the parsed value of FastForwardTTL
+ FastForwardTTL time.Duration `toml:"-"`
+ // FastForwardPath is the PathConfig to use for upstream Fast Forward Requests
+ FastForwardPath *PathConfig `toml:"-"`
+ // MaxTTL is the parsed value of MaxTTLSecs
+ MaxTTL time.Duration `toml:"-"`
+ // HTTPClient is the Client used by trickster to communicate with this origin
+ HTTPClient *http.Client `toml:"-"`
+ // CompressableTypes is the map version of CompressableTypeList for fast lookup
+ CompressableTypes map[string]bool `toml:"-"`
+}
+
+// CachingConfig is a collection of defining the Trickster Caching Behavior
+type CachingConfig struct {
+ // Name is the Name of the cache, taken from the Key in the Caches map[string]*CacheConfig
+ Name string `toml:"-"`
+ // Type represents the type of cache that we wish to use: "boltdb", "memory", "filesystem", or "redis"
+ CacheType string `toml:"cache_type"`
+ // Index provides options for the Cache Index
+ Index CacheIndexConfig `toml:"index"`
+ // Redis provides options for Redis caching
+ Redis RedisCacheConfig `toml:"redis"`
+ // Filesystem provides options for Filesystem caching
+ Filesystem FilesystemCacheConfig `toml:"filesystem"`
+ // BBolt provides options for BBolt caching
+ BBolt BBoltCacheConfig `toml:"bbolt"`
+ // Badger provides options for BadgerDB caching
+ Badger BadgerCacheConfig `toml:"badger"`
+
+ // Synthetic Values
+
+ // CacheTypeID represents the internal constant for the provided CacheType string
+ // and is automatically populated at startup
+ CacheTypeID CacheType `toml:"-"`
+}
+
+// CacheIndexConfig defines the operation of the Cache Indexer
+type CacheIndexConfig struct {
+ // ReapIntervalSecs defines how long the Cache Index reaper sleeps between reap cycles
+ ReapIntervalSecs int `toml:"reap_interval_secs"`
+ // FlushIntervalSecs sets how often the Cache Index saves its metadata to the cache from application memory
+ FlushIntervalSecs int `toml:"flush_interval_secs"`
+ // MaxSizeBytes indicates how large the cache can grow in bytes before the Index evicts
+ // least-recently-accessed items.
+ MaxSizeBytes int64 `toml:"max_size_bytes"`
+ // MaxSizeBackoffBytes indicates how far below max_size_bytes the cache size must be
+ // to complete a byte-size-based eviction exercise.
+ MaxSizeBackoffBytes int64 `toml:"max_size_backoff_bytes"`
+ // MaxSizeObjects indicates how large the cache can grow in objects before the Index
+ // evicts least-recently-accessed items.
+ MaxSizeObjects int64 `toml:"max_size_objects"`
+ // MaxSizeBackoffObjects indicates how far under max_size_objects the cache size must
+ // be to complete object-size-based eviction exercise.
+ MaxSizeBackoffObjects int64 `toml:"max_size_backoff_objects"`
+
+ ReapInterval time.Duration `toml:"-"`
+ FlushInterval time.Duration `toml:"-"`
+}
+
+// RedisCacheConfig is a collection of Configurations for Connecting to Redis
+type RedisCacheConfig struct {
+ // ClientType defines the type of Redis Client ("standard", "cluster", "sentinel")
+ ClientType string `toml:"client_type"`
+ // Protocol represents the connection method (e.g., "tcp", "unix", etc.)
+ Protocol string `toml:"protocol"`
+ // Endpoint represents FQDN:port or IPAddress:Port of the Redis Endpoint
+ Endpoint string `toml:"endpoint"`
+ // Endpoints represents FQDN:port or IPAddress:Port collection of a Redis Cluster or Sentinel Nodes
+ Endpoints []string `toml:"endpoints"`
+ // Password can be set when using password protected redis instance.
+ Password string `toml:"password"`
+ // SentinelMaster should be set when using Redis Sentinel to indicate the Master Node
+ SentinelMaster string `toml:"sentinel_master"`
+ // DB is the Database to be selected after connecting to the server.
+ DB int `toml:"db"`
+ // MaxRetries is the maximum number of retries before giving up on the command
+ MaxRetries int `toml:"max_retries"`
+ // MinRetryBackoffMS is the minimum backoff between each retry.
+ MinRetryBackoffMS int `toml:"min_retry_backoff_ms"`
+ // MaxRetryBackoffMS is the Maximum backoff between each retry.
+ MaxRetryBackoffMS int `toml:"max_retry_backoff_ms"`
+ // DialTimeoutMS is the timeout for establishing new connections.
+ DialTimeoutMS int `toml:"dial_timeout_ms"`
+ // ReadTimeoutMS is the timeout for socket reads. If reached, commands will fail with a timeout instead of blocking.
+ ReadTimeoutMS int `toml:"read_timeout_ms"`
+ // WriteTimeoutMS is the timeout for socket writes. If reached, commands will fail with a timeout instead of blocking.
+ WriteTimeoutMS int `toml:"write_timeout_ms"`
+ // PoolSize is the maximum number of socket connections.
+ PoolSize int `toml:"pool_size"`
+ // MinIdleConns is the minimum number of idle connections which is useful when establishing new connection is slow.
+ MinIdleConns int `toml:"min_idle_conns"`
+ // MaxConnAgeMS is the connection age at which client retires (closes) the connection.
+ MaxConnAgeMS int `toml:"max_conn_age_ms"`
+ // PoolTimeoutMS is the amount of time client waits for connection if all connections are busy before returning an error.
+ PoolTimeoutMS int `toml:"pool_timeout_ms"`
+ // IdleTimeoutMS is the amount of time after which client closes idle connections.
+ IdleTimeoutMS int `toml:"idle_timeout_ms"`
+ // IdleCheckFrequencyMS is the frequency of idle checks made by idle connections reaper.
+ IdleCheckFrequencyMS int `toml:"idle_check_frequency_ms"`
+}
+
+// BadgerCacheConfig is a collection of Configurations for storing cached data on the Filesystem in a Badger key-value store
+type BadgerCacheConfig struct {
+ // Directory represents the path on disk where the Badger database should store data
+ Directory string `toml:"directory"`
+ // ValueDirectory represents the path on disk where the Badger database will store its value log.
+ ValueDirectory string `toml:"value_directory"`
+}
+
+// BBoltCacheConfig is a collection of Configurations for storing cached data on the Filesystem
+type BBoltCacheConfig struct {
+ // Filename represents the filename (including path) of the BotlDB database
+ Filename string `toml:"filename"`
+ // Bucket represents the name of the bucket within BBolt under which Trickster's keys will be stored.
+ Bucket string `toml:"bucket"`
+}
+
+// FilesystemCacheConfig is a collection of Configurations for storing cached data on the Filesystem
+type FilesystemCacheConfig struct {
+ // CachePath represents the path on disk where our cache will live
+ CachePath string `toml:"cache_path"`
+}
+
+// FrontendConfig is a collection of configurations for the main http frontend for the application
+type FrontendConfig struct {
+ // ListenAddress is IP address for the main http listener for the application
+ ListenAddress string `toml:"listen_address"`
+ // ListenPort is TCP Port for the main http listener for the application
+ ListenPort int `toml:"listen_port"`
+ // TLSListenAddress is IP address for the tls http listener for the application
+ TLSListenAddress string `toml:"tls_listen_address"`
+ // TLSListenPort is the TCP Port for the tls http listener for the application
+ TLSListenPort int `toml:"tls_listen_port"`
+ // ConnectionsLimit indicates how many concurrent front end connections trickster will handle at any time
+ ConnectionsLimit int `toml:"connections_limit"`
+
+ // ServeTLS indicates whether to listen and serve on the TLS port, meaning
+ // at least one origin configuration has a valid certificate and key file configured.
+ ServeTLS bool `toml:"-"`
+}
+
+// LoggingConfig is a collection of Logging configurations
+type LoggingConfig struct {
+ // LogFile provides the filepath to the instances's logfile. Set as empty string to Log to Console
+ LogFile string `toml:"log_file"`
+ // LogLevel provides the most granular level (e.g., DEBUG, INFO, ERROR) to log
+ LogLevel string `toml:"log_level"`
+}
+
+// MetricsConfig is a collection of Metrics Collection configurations
+type MetricsConfig struct {
+ // ListenAddress is IP address from which the Application Metrics are available for pulling at /metrics
+ ListenAddress string `toml:"listen_address"`
+ // ListenPort is TCP Port from which the Application Metrics are available for pulling at /metrics
+ ListenPort int `toml:"listen_port"`
+}
+
+// NegativeCacheConfig is a collection of response codes and their TTLs
+type NegativeCacheConfig map[string]int
+
+// Clone returns an exact copy of a NegativeCacheConfig
+func (nc NegativeCacheConfig) Clone() NegativeCacheConfig {
+ nc2 := make(NegativeCacheConfig)
+ for k, v := range nc {
+ nc2[k] = v
+ }
+ return nc2
+}
+
+// NewConfig returns a Config initialized with default values.
+func NewConfig() *TricksterConfig {
+ return &TricksterConfig{
+ Caches: map[string]*CachingConfig{
+ "default": NewCacheConfig(),
+ },
+ Logging: &LoggingConfig{
+ LogFile: defaultLogFile,
+ LogLevel: defaultLogLevel,
+ },
+ Main: &MainConfig{
+ ConfigHandlerPath: defaultConfigHandlerPath,
+ PingHandlerPath: defaultPingHandlerPath,
+ },
+ Metrics: &MetricsConfig{
+ ListenPort: defaultMetricsListenPort,
+ },
+ Origins: map[string]*OriginConfig{
+ "default": NewOriginConfig(),
+ },
+ Frontend: &FrontendConfig{
+ ListenPort: defaultProxyListenPort,
+ },
+ NegativeCacheConfigs: map[string]NegativeCacheConfig{
+ "default": NewNegativeCacheConfig(),
+ },
+ }
+}
+
+// NewNegativeCacheConfig returns an empty NegativeCacheConfig
+func NewNegativeCacheConfig() NegativeCacheConfig {
+ return NegativeCacheConfig{}
+}
+
+// NewCacheConfig will return a pointer to an OriginConfig with the default configuration settings
+func NewCacheConfig() *CachingConfig {
+
+ return &CachingConfig{
+ CacheType: defaultCacheType,
+ CacheTypeID: defaultCacheTypeID,
+ Redis: RedisCacheConfig{ClientType: defaultRedisClientType, Protocol: defaultRedisProtocol, Endpoint: defaultRedisEndpoint, Endpoints: []string{defaultRedisEndpoint}},
+ Filesystem: FilesystemCacheConfig{CachePath: defaultCachePath},
+ BBolt: BBoltCacheConfig{Filename: defaultBBoltFile, Bucket: defaultBBoltBucket},
+ Badger: BadgerCacheConfig{Directory: defaultCachePath, ValueDirectory: defaultCachePath},
+ Index: CacheIndexConfig{
+ ReapIntervalSecs: defaultCacheIndexReap,
+ FlushIntervalSecs: defaultCacheIndexFlush,
+ MaxSizeBytes: defaultCacheMaxSizeBytes,
+ MaxSizeBackoffBytes: defaultMaxSizeBackoffBytes,
+ MaxSizeObjects: defaultMaxSizeObjects,
+ MaxSizeBackoffObjects: defaultMaxSizeBackoffObjects,
+ },
+ }
+}
+
+// NewOriginConfig will return a pointer to an OriginConfig with the default configuration settings
+func NewOriginConfig() *OriginConfig {
+ return &OriginConfig{
+ BackfillTolerance: defaultBackfillToleranceSecs,
+ BackfillToleranceSecs: defaultBackfillToleranceSecs,
+ CacheName: defaultOriginCacheName,
+ CacheKeyPrefix: "",
+ HealthCheckQuery: defaultHealthCheckQuery,
+ HealthCheckUpstreamPath: defaultHealthCheckPath,
+ HealthCheckVerb: defaultHealthCheckVerb,
+ HealthCheckHeaders: make(map[string]string),
+ KeepAliveTimeoutSecs: defaultKeepAliveTimeoutSecs,
+ MaxIdleConns: defaultMaxIdleConns,
+ NegativeCache: make(map[int]time.Duration),
+ NegativeCacheName: defaultOriginNegativeCacheName,
+ Paths: make(map[string]*PathConfig),
+ Timeout: time.Second * defaultOriginTimeoutSecs,
+ TimeoutSecs: defaultOriginTimeoutSecs,
+ TimeseriesEvictionMethod: defaultOriginTEM,
+ TimeseriesEvictionMethodName: defaultOriginTEMName,
+ TimeseriesRetention: defaultOriginTRF,
+ TimeseriesRetentionFactor: defaultOriginTRF,
+ TimeseriesTTLSecs: defaultTimeseriesTTLSecs,
+ FastForwardTTLSecs: defaultFastForwardTTLSecs,
+ TimeseriesTTL: defaultTimeseriesTTLSecs * time.Second,
+ FastForwardTTL: defaultFastForwardTTLSecs * time.Second,
+ MaxTTLSecs: defaultMaxTTLSecs,
+ MaxTTL: defaultMaxTTLSecs * time.Second,
+ RevalidationFactor: defaultRevalidationFactor,
+ MaxObjectSizeBytes: defaultMaxObjectSizeBytes,
+ TLS: &TLSConfig{},
+ CompressableTypeList: defaultCompressableTypes(),
+ }
+}
+
+// loadFile loads application configuration from a TOML-formatted file.
+func (c *TricksterConfig) loadFile() error {
+ md, err := toml.DecodeFile(Flags.ConfigPath, c)
+ if err != nil {
+ c.setDefaults(&toml.MetaData{})
+ return err
+ }
+ err = c.setDefaults(&md)
+ return err
+}
+
+func (c *TricksterConfig) setDefaults(metadata *toml.MetaData) error {
+
+ c.processOriginConfigs(metadata)
+ c.processCachingConfigs(metadata)
+ err := c.validateConfigMappings()
+ if err != nil {
+ return err
+ }
+
+ err = c.verifyTLSConfigs()
+
+ return err
+}
+
+var pathMembers = []string{"path", "match_type", "handler", "methods", "cache_key_params", "cache_key_headers", "default_ttl_secs",
+ "request_headers", "response_headers", "response_headers", "response_code", "response_body", "no_metrics", "progressive_collapsed_forwarding"}
+
+func (c *TricksterConfig) validateConfigMappings() error {
+ for k, oc := range c.Origins {
+ if _, ok := c.Caches[oc.CacheName]; !ok {
+ return fmt.Errorf("invalid cache name [%s] provided in origin config [%s]", oc.CacheName, k)
+ }
+ }
+ return nil
+}
+
+func (c *TricksterConfig) processOriginConfigs(metadata *toml.MetaData) {
+
+ c.activeCaches = make(map[string]bool)
+
+ for k, v := range c.Origins {
+
+ oc := NewOriginConfig()
+ oc.Name = k
+
+ if metadata.IsDefined("origins", k, "origin_type") {
+ oc.OriginType = v.OriginType
+ }
+
+ if metadata.IsDefined("origins", k, "is_default") {
+ oc.IsDefault = v.IsDefault
+ }
+ // If there is only one origin and is_default is not explicitly false, make it true
+ if len(c.Origins) == 1 && (!metadata.IsDefined("origins", k, "is_default")) {
+ oc.IsDefault = true
+ }
+
+ if metadata.IsDefined("origins", k, "require_tls") {
+ oc.RequireTLS = v.RequireTLS
+ }
+
+ if metadata.IsDefined("origins", k, "cache_name") {
+ oc.CacheName = v.CacheName
+ }
+ c.activeCaches[oc.CacheName] = true
+
+ if metadata.IsDefined("origins", k, "cache_key_prefix") {
+ oc.CacheKeyPrefix = v.CacheKeyPrefix
+ }
+
+ if metadata.IsDefined("origins", k, "origin_url") {
+ oc.OriginURL = v.OriginURL
+ }
+
+ if metadata.IsDefined("origins", k, "compressable_types") {
+ oc.CompressableTypeList = v.CompressableTypeList
+ }
+
+ if metadata.IsDefined("origins", k, "timeout_secs") {
+ oc.TimeoutSecs = v.TimeoutSecs
+ }
+
+ if metadata.IsDefined("origins", k, "max_idle_conns") {
+ oc.MaxIdleConns = v.MaxIdleConns
+ }
+
+ if metadata.IsDefined("origins", k, "keep_alive_timeout_secs") {
+ oc.KeepAliveTimeoutSecs = v.KeepAliveTimeoutSecs
+ }
+
+ if metadata.IsDefined("origins", k, "timeseries_retention_factor") {
+ oc.TimeseriesRetentionFactor = v.TimeseriesRetentionFactor
+ }
+
+ if metadata.IsDefined("origins", k, "timeseries_eviction_method") {
+ oc.TimeseriesEvictionMethodName = strings.ToLower(v.TimeseriesEvictionMethodName)
+ if p, ok := timeseriesEvictionMethodNames[oc.TimeseriesEvictionMethodName]; ok {
+ oc.TimeseriesEvictionMethod = p
+ }
+ }
+
+ if metadata.IsDefined("origins", k, "timeseries_ttl_secs") {
+ oc.TimeseriesTTLSecs = v.TimeseriesTTLSecs
+ }
+
+ if metadata.IsDefined("origins", k, "max_ttl_secs") {
+ oc.MaxTTLSecs = v.MaxTTLSecs
+ }
+
+ if metadata.IsDefined("origins", k, "fastforward_ttl_secs") {
+ oc.FastForwardTTLSecs = v.FastForwardTTLSecs
+ }
+
+ if metadata.IsDefined("origins", k, "fast_forward_disable") {
+ oc.FastForwardDisable = v.FastForwardDisable
+ }
+
+ if metadata.IsDefined("origins", k, "backfill_tolerance_secs") {
+ oc.BackfillToleranceSecs = v.BackfillToleranceSecs
+ }
+
+ if metadata.IsDefined("origins", k, "paths") {
+ var j = 0
+ for l, p := range v.Paths {
+ if len(p.Methods) == 0 {
+ p.Methods = []string{http.MethodGet, http.MethodHead}
+ }
+ p.custom = make([]string, 0)
+ for _, pm := range pathMembers {
+ if metadata.IsDefined("origins", k, "paths", l, pm) {
+ p.custom = append(p.custom, pm)
+ }
+ }
+ if metadata.IsDefined("origins", k, "paths", l, "response_body") {
+ p.ResponseBodyBytes = []byte(p.ResponseBody)
+ p.HasCustomResponseBody = true
+ }
+
+ if mt, ok := pathMatchTypeNames[strings.ToLower(p.MatchTypeName)]; ok {
+ p.MatchType = mt
+ p.MatchTypeName = p.MatchType.String()
+ } else {
+ p.MatchType = PathMatchTypeExact
+ p.MatchTypeName = p.MatchType.String()
+ }
+ oc.Paths[p.Path+"-"+strings.Join(p.Methods, "-")] = p
+ j++
+ }
+ }
+
+ if metadata.IsDefined("origins", k, "negative_cache_name") {
+ oc.NegativeCacheName = v.NegativeCacheName
+ }
+
+ if metadata.IsDefined("origins", k, "health_check_upstream_path") {
+ oc.HealthCheckUpstreamPath = v.HealthCheckUpstreamPath
+ }
+
+ if metadata.IsDefined("origins", k, "health_check_verb") {
+ oc.HealthCheckVerb = v.HealthCheckVerb
+ }
+
+ if metadata.IsDefined("origins", k, "health_check_query") {
+ oc.HealthCheckQuery = v.HealthCheckQuery
+ }
+
+ if metadata.IsDefined("origins", k, "health_check_headers") {
+ oc.HealthCheckHeaders = v.HealthCheckHeaders
+ }
+
+ if metadata.IsDefined("origins", k, "max_object_size_bytes") {
+ oc.MaxObjectSizeBytes = v.MaxObjectSizeBytes
+ }
+
+ if metadata.IsDefined("origins", k, "revalidation_factor") {
+ oc.RevalidationFactor = v.RevalidationFactor
+ }
+
+ if metadata.IsDefined("origins", k, "multipart_ranges_disabled") {
+ oc.MultipartRangesDisabled = v.MultipartRangesDisabled
+ }
+
+ if metadata.IsDefined("origins", k, "dearticulate_upstream_ranges") {
+ oc.DearticulateUpstreamRanges = v.DearticulateUpstreamRanges
+ }
+
+ if metadata.IsDefined("origins", k, "tls") {
+ oc.TLS = &TLSConfig{
+ InsecureSkipVerify: v.TLS.InsecureSkipVerify,
+ CertificateAuthorityPaths: v.TLS.CertificateAuthorityPaths,
+ PrivateKeyPath: v.TLS.PrivateKeyPath,
+ FullChainCertPath: v.TLS.FullChainCertPath,
+ ClientCertPath: v.TLS.ClientCertPath,
+ ClientKeyPath: v.TLS.ClientKeyPath,
+ }
+ }
+
+ c.Origins[k] = oc
+ }
+}
+
+func (c *TricksterConfig) processCachingConfigs(metadata *toml.MetaData) {
+
+ // setCachingDefaults assumes that processOriginConfigs was just ran
+
+ for k, v := range c.Caches {
+
+ if _, ok := c.activeCaches[k]; !ok {
+ // a configured cache was not used by any origin. don't even instantiate it
+ delete(c.Caches, k)
+ continue
+ }
+
+ cc := NewCacheConfig()
+ cc.Name = k
+
+ if metadata.IsDefined("caches", k, "cache_type") {
+ cc.CacheType = strings.ToLower(v.CacheType)
+ if n, ok := CacheTypeNames[cc.CacheType]; ok {
+ cc.CacheTypeID = n
+ }
+ }
+
+ if metadata.IsDefined("caches", k, "index", "reap_interval_secs") {
+ cc.Index.ReapIntervalSecs = v.Index.ReapIntervalSecs
+ }
+
+ if metadata.IsDefined("caches", k, "index", "flush_interval_secs") {
+ cc.Index.FlushIntervalSecs = v.Index.FlushIntervalSecs
+ }
+
+ if metadata.IsDefined("caches", k, "index", "max_size_bytes") {
+ cc.Index.MaxSizeBytes = v.Index.MaxSizeBytes
+ }
+
+ if metadata.IsDefined("caches", k, "index", "max_size_backoff_bytes") {
+ cc.Index.MaxSizeBackoffBytes = v.Index.MaxSizeBackoffBytes
+ }
+
+ if metadata.IsDefined("caches", k, "index", "max_size_objects") {
+ cc.Index.MaxSizeObjects = v.Index.MaxSizeObjects
+ }
+
+ if metadata.IsDefined("caches", k, "index", "max_size_backoff_objects") {
+ cc.Index.MaxSizeBackoffObjects = v.Index.MaxSizeBackoffObjects
+ }
+
+ if cc.CacheTypeID == CacheTypeRedis {
+
+ var hasEndpoint, hasEndpoints bool
+
+ ct := strings.ToLower(v.Redis.ClientType)
+ if metadata.IsDefined("caches", k, "redis", "client_type") {
+ cc.Redis.ClientType = ct
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "protocol") {
+ cc.Redis.Protocol = v.Redis.Protocol
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "endpoint") {
+ cc.Redis.Endpoint = v.Redis.Endpoint
+ hasEndpoint = true
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "endpoints") {
+ cc.Redis.Endpoints = v.Redis.Endpoints
+ hasEndpoints = true
+ }
+
+ if cc.Redis.ClientType == "standard" {
+ if hasEndpoints && !hasEndpoint {
+ LoaderWarnings = append(LoaderWarnings, "'standard' redis type configured, but 'endpoints' value is provided instead of 'endpoint'")
+ }
+ } else {
+ if hasEndpoint && !hasEndpoints {
+ LoaderWarnings = append(LoaderWarnings, fmt.Sprintf("'%s' redis type configured, but 'endpoint' value is provided instead of 'endpoints'", cc.Redis.ClientType))
+ }
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "sentinel_master") {
+ cc.Redis.SentinelMaster = v.Redis.SentinelMaster
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "password") {
+ cc.Redis.Password = v.Redis.Password
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "db") {
+ cc.Redis.DB = v.Redis.DB
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "max_retries") {
+ cc.Redis.MaxRetries = v.Redis.MaxRetries
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "min_retry_backoff_ms") {
+ cc.Redis.MinRetryBackoffMS = v.Redis.MinRetryBackoffMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "max_retry_backoff_ms") {
+ cc.Redis.MaxRetryBackoffMS = v.Redis.MaxRetryBackoffMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "dial_timeout_ms") {
+ cc.Redis.DialTimeoutMS = v.Redis.DialTimeoutMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "read_timeout_ms") {
+ cc.Redis.ReadTimeoutMS = v.Redis.ReadTimeoutMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "write_timeout_ms") {
+ cc.Redis.WriteTimeoutMS = v.Redis.WriteTimeoutMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "pool_size") {
+ cc.Redis.PoolSize = v.Redis.PoolSize
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "min_idle_conns") {
+ cc.Redis.MinIdleConns = v.Redis.MinIdleConns
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "max_conn_age_ms") {
+ cc.Redis.MaxConnAgeMS = v.Redis.MaxConnAgeMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "pool_timeout_ms") {
+ cc.Redis.PoolTimeoutMS = v.Redis.PoolTimeoutMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "idle_timeout_ms") {
+ cc.Redis.IdleTimeoutMS = v.Redis.IdleTimeoutMS
+ }
+
+ if metadata.IsDefined("caches", k, "redis", "idle_check_frequency_ms") {
+ cc.Redis.IdleCheckFrequencyMS = v.Redis.IdleCheckFrequencyMS
+ }
+ }
+
+ if metadata.IsDefined("caches", k, "filesystem", "cache_path") {
+ cc.Filesystem.CachePath = v.Filesystem.CachePath
+ }
+
+ if metadata.IsDefined("caches", k, "bbolt", "filename") {
+ cc.BBolt.Filename = v.BBolt.Filename
+ }
+
+ if metadata.IsDefined("caches", k, "bbolt", "bucket") {
+ cc.BBolt.Bucket = v.BBolt.Bucket
+ }
+
+ if metadata.IsDefined("caches", k, "badger", "directory") {
+ cc.Badger.Directory = v.Badger.Directory
+ }
+
+ if metadata.IsDefined("caches", k, "badger", "value_directory") {
+ cc.Badger.ValueDirectory = v.Badger.ValueDirectory
+ }
+
+ c.Caches[k] = cc
+ }
+}
+
+func (c *TricksterConfig) copy() *TricksterConfig {
+
+ nc := NewConfig()
+ delete(nc.Caches, "default")
+ delete(nc.Origins, "default")
+
+ nc.Main.ConfigHandlerPath = c.Main.ConfigHandlerPath
+ nc.Main.InstanceID = c.Main.InstanceID
+ nc.Main.PingHandlerPath = c.Main.PingHandlerPath
+
+ nc.Logging.LogFile = c.Logging.LogFile
+ nc.Logging.LogLevel = c.Logging.LogLevel
+
+ nc.Metrics.ListenAddress = c.Metrics.ListenAddress
+ nc.Metrics.ListenPort = c.Metrics.ListenPort
+
+ nc.Frontend.ListenAddress = c.Frontend.ListenAddress
+ nc.Frontend.ListenPort = c.Frontend.ListenPort
+ nc.Frontend.TLSListenAddress = c.Frontend.TLSListenAddress
+ nc.Frontend.TLSListenPort = c.Frontend.TLSListenPort
+ nc.Frontend.ConnectionsLimit = c.Frontend.ConnectionsLimit
+ nc.Frontend.ServeTLS = c.Frontend.ServeTLS
+
+ for k, v := range c.Origins {
+ nc.Origins[k] = v.Clone()
+ }
+
+ for k, v := range c.Caches {
+ nc.Caches[k] = v.Clone()
+ }
+
+ for k, v := range c.NegativeCacheConfigs {
+ nc.NegativeCacheConfigs[k] = v.Clone()
+ }
+
+ return nc
+}
+
+func (c *TricksterConfig) String() string {
+ cp := c.copy()
+
+ // the toml library will panic if the Handler is assigned,
+ // even though this field is annotated as skip ("-") in the prototype
+ // so we'll iterate the paths and set to nil the Handler (in our local copy only)
+ if cp.Origins != nil {
+ for _, v := range cp.Origins {
+ if v != nil {
+ for _, w := range v.Paths {
+ w.Handler = nil
+ w.KeyHasher = nil
+ }
+ }
+ // also strip out potentially sensitive headers
+ hideAuthorizationCredentials(v.HealthCheckHeaders)
+
+ if v.Paths != nil {
+ for _, p := range v.Paths {
+ hideAuthorizationCredentials(p.RequestHeaders)
+ hideAuthorizationCredentials(p.ResponseHeaders)
+ }
+ }
+ }
+ }
+
+ // strip Redis password
+ for k, v := range cp.Caches {
+ if v != nil && cp.Caches[k].Redis.Password != "" {
+ cp.Caches[k].Redis.Password = "*****"
+ }
+ }
+
+ var buf bytes.Buffer
+ e := toml.NewEncoder(&buf)
+ e.Encode(cp)
+ return buf.String()
+}
+
+var sensitiveCredentials = map[string]bool{headers.NameAuthorization: true}
+
+func hideAuthorizationCredentials(headers map[string]string) {
+ // strip Authorization Headers
+ for k := range headers {
+ if _, ok := sensitiveCredentials[k]; ok {
+ headers[k] = "*****"
+ }
+ }
+}
+
+// Clone returns an exact copy of an *OriginConfig
+func (oc *OriginConfig) Clone() *OriginConfig {
+
+ o := &OriginConfig{}
+ o.DearticulateUpstreamRanges = oc.DearticulateUpstreamRanges
+ o.BackfillTolerance = oc.BackfillTolerance
+ o.BackfillToleranceSecs = oc.BackfillToleranceSecs
+ o.CacheName = oc.CacheName
+ o.CacheKeyPrefix = oc.CacheKeyPrefix
+ o.FastForwardDisable = oc.FastForwardDisable
+ o.FastForwardTTL = oc.FastForwardTTL
+ o.FastForwardTTLSecs = oc.FastForwardTTLSecs
+ o.HealthCheckUpstreamPath = oc.HealthCheckUpstreamPath
+ o.HealthCheckVerb = oc.HealthCheckVerb
+ o.HealthCheckQuery = oc.HealthCheckQuery
+ o.Host = oc.Host
+ o.Name = oc.Name
+ o.IsDefault = oc.IsDefault
+ o.KeepAliveTimeoutSecs = oc.KeepAliveTimeoutSecs
+ o.MaxIdleConns = oc.MaxIdleConns
+ o.MaxTTLSecs = oc.MaxTTLSecs
+ o.MaxTTL = oc.MaxTTL
+ o.MaxObjectSizeBytes = oc.MaxObjectSizeBytes
+ o.MultipartRangesDisabled = oc.MultipartRangesDisabled
+ o.OriginType = oc.OriginType
+ o.OriginURL = oc.OriginURL
+ o.PathPrefix = oc.PathPrefix
+ o.RevalidationFactor = oc.RevalidationFactor
+ o.Scheme = oc.Scheme
+ o.Timeout = oc.Timeout
+ o.TimeoutSecs = oc.TimeoutSecs
+ o.TimeseriesRetention = oc.TimeseriesRetention
+ o.TimeseriesRetentionFactor = oc.TimeseriesRetentionFactor
+ o.TimeseriesEvictionMethodName = oc.TimeseriesEvictionMethodName
+ o.TimeseriesEvictionMethod = oc.TimeseriesEvictionMethod
+ o.TimeseriesTTL = oc.TimeseriesTTL
+ o.TimeseriesTTLSecs = oc.TimeseriesTTLSecs
+ o.ValueRetention = oc.ValueRetention
+
+ if oc.CompressableTypeList != nil {
+ o.CompressableTypeList = make([]string, len(oc.CompressableTypeList))
+ copy(o.CompressableTypeList, oc.CompressableTypeList)
+ }
+
+ if oc.CompressableTypes != nil {
+ o.CompressableTypes = make(map[string]bool)
+ for k := range oc.CompressableTypes {
+ o.CompressableTypes[k] = true
+ }
+ }
+
+ o.HealthCheckHeaders = make(map[string]string)
+ for k, v := range oc.HealthCheckHeaders {
+ o.HealthCheckHeaders[k] = v
+ }
+
+ o.Paths = make(map[string]*PathConfig)
+ for l, p := range oc.Paths {
+ o.Paths[l] = p.Clone()
+ }
+
+ o.NegativeCacheName = oc.NegativeCacheName
+ if oc.NegativeCache != nil {
+ m := make(map[int]time.Duration)
+ for c, t := range oc.NegativeCache {
+ m[c] = t
+ }
+ o.NegativeCache = m
+ }
+
+ if oc.TLS != nil {
+ o.TLS = oc.TLS.Clone()
+ }
+ o.RequireTLS = oc.RequireTLS
+
+ if oc.FastForwardPath != nil {
+ o.FastForwardPath = oc.FastForwardPath.Clone()
+ }
+
+ return o
+
+}
+
+// Clone returns an exact copy of a *CachingConfig
+func (cc *CachingConfig) Clone() *CachingConfig {
+
+ c := NewCacheConfig()
+ c.Name = cc.Name
+ c.CacheType = cc.CacheType
+ c.CacheTypeID = cc.CacheTypeID
+
+ c.Index.FlushInterval = cc.Index.FlushInterval
+ c.Index.FlushIntervalSecs = cc.Index.FlushIntervalSecs
+ c.Index.MaxSizeBackoffBytes = cc.Index.MaxSizeBackoffBytes
+ c.Index.MaxSizeBackoffObjects = cc.Index.MaxSizeBackoffObjects
+ c.Index.MaxSizeBytes = cc.Index.MaxSizeBytes
+ c.Index.MaxSizeObjects = cc.Index.MaxSizeObjects
+ c.Index.ReapInterval = cc.Index.ReapInterval
+ c.Index.ReapIntervalSecs = cc.Index.ReapIntervalSecs
+
+ c.Badger.Directory = cc.Badger.Directory
+ c.Badger.ValueDirectory = cc.Badger.ValueDirectory
+
+ c.Filesystem.CachePath = cc.Filesystem.CachePath
+
+ c.BBolt.Bucket = cc.BBolt.Bucket
+ c.BBolt.Filename = cc.BBolt.Filename
+
+ c.Redis.ClientType = cc.Redis.ClientType
+ c.Redis.DB = cc.Redis.DB
+ c.Redis.DialTimeoutMS = cc.Redis.DialTimeoutMS
+ c.Redis.Endpoint = cc.Redis.Endpoint
+ c.Redis.Endpoints = cc.Redis.Endpoints
+ c.Redis.IdleCheckFrequencyMS = cc.Redis.IdleCheckFrequencyMS
+ c.Redis.IdleTimeoutMS = cc.Redis.IdleTimeoutMS
+ c.Redis.MaxConnAgeMS = cc.Redis.MaxConnAgeMS
+ c.Redis.MaxRetries = cc.Redis.MaxRetries
+ c.Redis.MaxRetryBackoffMS = cc.Redis.MaxRetryBackoffMS
+ c.Redis.MinIdleConns = cc.Redis.MinIdleConns
+ c.Redis.MinRetryBackoffMS = cc.Redis.MinRetryBackoffMS
+ c.Redis.Password = cc.Redis.Password
+ c.Redis.PoolSize = cc.Redis.PoolSize
+ c.Redis.PoolTimeoutMS = cc.Redis.PoolTimeoutMS
+ c.Redis.Protocol = cc.Redis.Protocol
+ c.Redis.ReadTimeoutMS = cc.Redis.ReadTimeoutMS
+ c.Redis.SentinelMaster = cc.Redis.SentinelMaster
+ c.Redis.WriteTimeoutMS = cc.Redis.WriteTimeoutMS
+
+ return c
+
+}
diff --git a/internal/config/config_test.go b/internal/config/config_test.go
new file mode 100644
index 000000000..3e627f835
--- /dev/null
+++ b/internal/config/config_test.go
@@ -0,0 +1,64 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/proxy/headers"
+)
+
+func TestCopy(t *testing.T) {
+ c1 := NewConfig()
+
+ oc := c1.Origins["default"]
+ c1.NegativeCacheConfigs["default"]["404"] = 10
+
+ oc.CompressableTypeList = []string{"text/plain"}
+ oc.CompressableTypes = map[string]bool{"text/plain": true}
+ oc.NegativeCacheName = "default"
+ oc.NegativeCache = map[int]time.Duration{404: time.Duration(10) * time.Second}
+ oc.FastForwardPath = NewPathConfig()
+ oc.TLS = &TLSConfig{CertificateAuthorityPaths: []string{"foo"}}
+ oc.HealthCheckHeaders = map[string]string{headers.NameAuthorization: "Basic SomeHash"}
+
+ c2 := c1.copy()
+ if !reflect.DeepEqual(c1, c2) {
+ t.Errorf("copy mistmatch")
+ }
+}
+
+func TestString(t *testing.T) {
+ c1 := NewConfig()
+
+ c1.Origins["default"].Paths["test"] = &PathConfig{}
+
+ c1.Caches["default"].Redis.Password = "plaintext-password"
+
+ s := c1.String()
+ if !strings.Contains(s, `password = "*****"`) {
+ t.Errorf("missing password mask: %s", "*****")
+ }
+}
+
+func TestHideAuthorizationCredentials(t *testing.T) {
+ hdrs := map[string]string{headers.NameAuthorization: "Basic SomeHash"}
+ hideAuthorizationCredentials(hdrs)
+ if hdrs[headers.NameAuthorization] != "*****" {
+ t.Errorf("expected '*****' got '%s'", hdrs[headers.NameAuthorization])
+ }
+}
diff --git a/internal/config/defaults.go b/internal/config/defaults.go
new file mode 100644
index 000000000..51d34eebf
--- /dev/null
+++ b/internal/config/defaults.go
@@ -0,0 +1,81 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+const (
+ defaultLogFile = ""
+ defaultLogLevel = "INFO"
+
+ defaultProxyListenPort = 9090
+ defaultProxyListenAddress = ""
+
+ defaultMetricsListenPort = 8082
+ defaultMetricsListenAddress = ""
+
+ defaultCacheType = "memory"
+ defaultCacheTypeID = CacheTypeMemory
+
+ defaultTimeseriesTTLSecs = 21600
+ defaultFastForwardTTLSecs = 15
+ defaultMaxTTLSecs = 86400
+ defaultRevalidationFactor = 2
+
+ defaultCachePath = "/tmp/trickster"
+
+ defaultRedisClientType = "standard"
+ defaultRedisProtocol = "tcp"
+ defaultRedisEndpoint = "redis:6379"
+
+ defaultBBoltFile = "trickster.db"
+ defaultBBoltBucket = "trickster"
+
+ defaultCacheIndexReap = 3
+ defaultCacheIndexFlush = 5
+ defaultCacheMaxSizeBytes = 536870912
+ defaultMaxSizeBackoffBytes = 16777216
+ defaultMaxSizeObjects = 0
+ defaultMaxSizeBackoffObjects = 100
+ defaultMaxObjectSizeBytes = 524288
+
+ defaultOriginTRF = 1024
+ defaultOriginTEM = EvictionMethodOldest
+ defaultOriginTEMName = "oldest"
+ defaultOriginTimeoutSecs = 180
+ defaultOriginCacheName = "default"
+ defaultOriginNegativeCacheName = "default"
+ defaultBackfillToleranceSecs = 0
+ defaultKeepAliveTimeoutSecs = 300
+ defaultMaxIdleConns = 20
+
+ defaultHealthCheckPath = "-"
+ defaultHealthCheckQuery = "-"
+ defaultHealthCheckVerb = "-"
+
+ defaultConfigHandlerPath = "/trickster/config"
+ defaultPingHandlerPath = "/trickster/ping"
+)
+
+func defaultCompressableTypes() []string {
+ return []string{
+ "text/html",
+ "text/javascript",
+ "text/css",
+ "text/plain",
+ "text/xml",
+ "text/json",
+ "application/json",
+ "application/javascript",
+ "application/xml",
+ }
+}
diff --git a/internal/config/env.go b/internal/config/env.go
new file mode 100644
index 000000000..02c035730
--- /dev/null
+++ b/internal/config/env.go
@@ -0,0 +1,59 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "os"
+ "strconv"
+)
+
+const (
+ // Environment variables
+ evOriginURL = "TRK_ORIGIN_URL"
+ evOriginType = "TRK_ORIGIN_TYPE"
+ evProxyPort = "TRK_PROXY_PORT"
+ evMetricsPort = "TRK_METRICS_PORT"
+ evLogLevel = "TRK_LOG_LEVEL"
+)
+
+func (c *TricksterConfig) loadEnvVars() {
+ // Origin
+ if x := os.Getenv(evOriginURL); x != "" {
+ providedOriginURL = x
+ }
+
+ if x := os.Getenv(evOriginType); x != "" {
+ providedOriginType = x
+ }
+
+ // Proxy Port
+ if x := os.Getenv(evProxyPort); x != "" {
+ if y, err := strconv.ParseInt(x, 10, 64); err == nil {
+ c.Frontend.ListenPort = int(y)
+ }
+ }
+
+ // Metrics Port
+ if x := os.Getenv(evMetricsPort); x != "" {
+ if y, err := strconv.ParseInt(x, 10, 64); err == nil {
+ c.Metrics.ListenPort = int(y)
+ }
+ }
+
+ // LogLevel
+ if x := os.Getenv(evLogLevel); x != "" {
+ c.Logging.LogLevel = x
+ }
+
+}
diff --git a/internal/config/env_test.go b/internal/config/env_test.go
new file mode 100644
index 000000000..9e06a6a06
--- /dev/null
+++ b/internal/config/env_test.go
@@ -0,0 +1,71 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "os"
+ "strings"
+ "testing"
+)
+
+func TestLoadEnvVars(t *testing.T) {
+
+ os.Setenv(evOriginURL, "http://1.1.1.1:9090/some/path")
+ os.Setenv(evOriginType, "testing")
+ os.Setenv(evProxyPort, "4001")
+ os.Setenv(evMetricsPort, "4002")
+ os.Setenv(evLogLevel, "info")
+
+ a := []string{}
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ d := Origins["default"]
+ if d.OriginType != "testing" {
+ t.Errorf("expected %s got %s", "testing", d.OriginType)
+ }
+
+ if Frontend.ListenPort != 4001 {
+ t.Errorf("expected %d got %d", 4001, Frontend.ListenPort)
+ }
+
+ if Metrics.ListenPort != 4002 {
+ t.Errorf("expected %d got %d", 4002, Metrics.ListenPort)
+ }
+
+ if d.Scheme != "http" {
+ t.Errorf("expected %s got %s", "http", d.Scheme)
+ }
+
+ if d.Host != "1.1.1.1:9090" {
+ t.Errorf("expected %s got %s", "1.1.1.1:9090", d.Host)
+ }
+
+ if d.PathPrefix != "/some/path" {
+ t.Errorf("expected %s got %s", "/some/path", d.PathPrefix)
+ }
+
+ if strings.ToUpper(Logging.LogLevel) != "INFO" {
+ t.Errorf("expected %s got %s", "INFO", Logging.LogLevel)
+ }
+
+ os.Unsetenv(evOriginURL)
+ os.Unsetenv(evOriginType)
+ os.Unsetenv(evProxyPort)
+ os.Unsetenv(evMetricsPort)
+ os.Unsetenv(evLogLevel)
+
+}
diff --git a/internal/config/flags.go b/internal/config/flags.go
new file mode 100644
index 000000000..8b44a2c6f
--- /dev/null
+++ b/internal/config/flags.go
@@ -0,0 +1,90 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "flag"
+)
+
+const (
+ // Command-line flags
+ cfConfig = "config"
+ cfVersion = "version"
+ cfLogLevel = "log-level"
+ cfInstanceID = "instance-id"
+ cfOrigin = "origin-url"
+ cfOriginType = "origin-type"
+ cfProxyPort = "proxy-port"
+ cfMetricsPort = "metrics-port"
+
+ // DefaultConfigPath defines the default location of the Trickster config file
+ DefaultConfigPath = "/etc/trickster/trickster.conf"
+)
+
+// TricksterFlags holds the values for whitelisted flags
+type TricksterFlags struct {
+ PrintVersion bool
+ ConfigPath string
+ customPath bool
+ Origin string
+ OriginType string
+ ProxyListenPort int
+ MetricsListenPort int
+ LogLevel string
+ InstanceID int
+}
+
+// loadFlags loads configuration from command line flags.
+func (c *TricksterConfig) parseFlags(applicationName string, arguments []string) {
+
+ Flags = TricksterFlags{}
+
+ f := flag.NewFlagSet(applicationName, flag.ExitOnError)
+ f.BoolVar(&Flags.PrintVersion, cfVersion, false, "Prints trickster version")
+ f.StringVar(&Flags.ConfigPath, cfConfig, "", "Path to Trickster Config File")
+ f.StringVar(&Flags.LogLevel, cfLogLevel, "", "Level of Logging to use (debug, info, warn, error)")
+ f.IntVar(&Flags.InstanceID, cfInstanceID, 0, "Instance ID is for running multiple Trickster processes from the same config while logging to their own files.")
+ f.StringVar(&Flags.Origin, cfOrigin, "", "URL to the Origin. Enter it like you would in grafana, e.g., http://prometheus:9090")
+ f.StringVar(&Flags.OriginType, cfOriginType, "", "Type of origin (prometheus, influxdb)")
+ f.IntVar(&Flags.ProxyListenPort, cfProxyPort, 0, "Port that the primary Proxy server will listen on.")
+ f.IntVar(&Flags.MetricsListenPort, cfMetricsPort, 0, "Port that the /metrics endpoint will listen on.")
+ f.Parse(arguments)
+
+ if Flags.ConfigPath != "" {
+ Flags.customPath = true
+ } else {
+ Flags.ConfigPath = DefaultConfigPath
+ }
+}
+
+func (c *TricksterConfig) loadFlags() {
+ if len(Flags.Origin) > 0 {
+ providedOriginURL = Flags.Origin
+ }
+ if len(Flags.OriginType) > 0 {
+ providedOriginType = Flags.OriginType
+ }
+ if Flags.ProxyListenPort > 0 {
+ c.Frontend.ListenPort = Flags.ProxyListenPort
+ }
+ if Flags.MetricsListenPort > 0 {
+ c.Metrics.ListenPort = Flags.MetricsListenPort
+ }
+ if Flags.LogLevel != "" {
+ c.Logging.LogLevel = Flags.LogLevel
+ }
+ if Flags.InstanceID > 0 {
+ c.Main.InstanceID = Flags.InstanceID
+ }
+}
diff --git a/flags_test.go b/internal/config/flags_test.go
similarity index 65%
rename from flags_test.go
rename to internal/config/flags_test.go
index 6e4a0de16..87b617e8b 100644
--- a/flags_test.go
+++ b/internal/config/flags_test.go
@@ -11,45 +11,43 @@
* limitations under the License.
*/
-package main
+package config
import (
"testing"
)
func TestLoadFlags(t *testing.T) {
- c := Config{}
+ c := NewConfig()
a := []string{
- "-origin",
+ "-origin-url",
"http://prometheus.example.com:9090",
"-proxy-port",
"9091",
"-metrics-port",
"9092",
+ "-origin-type",
+ "prometheus",
+ "-log-level",
+ "info",
+ "-instance-id",
+ "1",
}
// it should read command line flags
- loadFlags(&c, a)
+ c.parseFlags("trickster-test", a)
+ c.loadFlags()
- if c.DefaultOriginURL != a[1] {
- t.Errorf("wanted \"%s\". got \"%s\".", a[1], c.DefaultOriginURL)
+ if providedOriginURL != a[1] {
+ t.Errorf("wanted \"%s\". got \"%s\".", a[1], providedOriginURL)
}
- if c.ProxyServer.ListenPort != 9091 {
- t.Errorf("wanted \"%d\". got \"%d\".", 9091, c.ProxyServer.ListenPort)
+ if providedOriginType != a[7] {
+ t.Errorf("wanted \"%s\". got \"%s\".", a[1], providedOriginType)
+ }
+ if c.Frontend.ListenPort != 9091 {
+ t.Errorf("wanted \"%d\". got \"%d\".", 9091, c.Frontend.ListenPort)
}
if c.Metrics.ListenPort != 9092 {
t.Errorf("wanted \"%d\". got \"%d\".", 9092, c.Metrics.ListenPort)
}
}
-
-func TestLoadConfiguration(t *testing.T) {
- c := Config{}
- a := []string{}
-
- // it should not error if config path is not set
- err := loadConfiguration(&c, a)
-
- if err != nil {
- t.Error(err)
- }
-}
diff --git a/internal/config/loader.go b/internal/config/loader.go
new file mode 100644
index 000000000..9e77a0429
--- /dev/null
+++ b/internal/config/loader.go
@@ -0,0 +1,171 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "fmt"
+ "net/url"
+ "strconv"
+ "strings"
+ "time"
+)
+
+// Load returns the Application Configuration, starting with a default config,
+// then overriding with any provided config file, then env vars, and finally flags
+func Load(applicationName string, applicationVersion string, arguments []string) error {
+
+ providedOriginURL = ""
+ providedOriginType = ""
+
+ LoaderWarnings = make([]string, 0)
+
+ c := NewConfig()
+ c.parseFlags(applicationName, arguments) // Parse here to get config file path and version flags
+ if Flags.PrintVersion {
+ return nil
+ }
+ if err := c.loadFile(); err != nil && Flags.customPath {
+ // a user-provided path couldn't be loaded. return the error for the application to handle
+ return err
+ }
+
+ c.loadEnvVars()
+ c.loadFlags() // load parsed flags to override file and envs
+
+ // set the default origin url from the flags
+ if d, ok := c.Origins["default"]; ok {
+ if providedOriginURL != "" {
+ url, err := url.Parse(providedOriginURL)
+ if err != nil {
+ return err
+ }
+ if providedOriginType != "" {
+ d.OriginType = providedOriginType
+ }
+ d.OriginURL = providedOriginURL
+ d.Scheme = url.Scheme
+ d.Host = url.Host
+ d.PathPrefix = url.Path
+ }
+ // If the user has configured their own origins, and one of them is not "default"
+ // then Trickster will not use the auto-created default origin
+ if d.OriginURL == "" {
+ delete(c.Origins, "default")
+ }
+
+ if providedOriginType != "" {
+ d.OriginType = providedOriginType
+ }
+ }
+
+ if len(c.Origins) == 0 {
+ return fmt.Errorf("no valid origins configured%s", "")
+ }
+
+ Config = c
+ Main = c.Main
+ Origins = c.Origins
+ Caches = c.Caches
+ Frontend = c.Frontend
+ Logging = c.Logging
+ Metrics = c.Metrics
+ NegativeCacheConfigs = c.NegativeCacheConfigs
+
+ for k, n := range NegativeCacheConfigs {
+ for c := range n {
+ ci, err := strconv.Atoi(c)
+ if err != nil {
+ return fmt.Errorf(`invalid negative cache config in %s: %s is not a valid status code`, k, c)
+ }
+ if ci < 400 || ci >= 600 {
+ return fmt.Errorf(`invalid negative cache config in %s: %s is not a valid status code`, k, c)
+ }
+ }
+ }
+
+ for k, o := range c.Origins {
+
+ if o.OriginURL == "" {
+ return fmt.Errorf(`missing origin-url for origin "%s"`, k)
+ }
+
+ url, err := url.Parse(o.OriginURL)
+ if err != nil {
+ return err
+ }
+
+ if o.OriginType == "" {
+ return fmt.Errorf(`missing origin-type for origin "%s"`, k)
+ }
+
+ if strings.HasSuffix(url.Path, "/") {
+ url.Path = url.Path[0 : len(url.Path)-1]
+ }
+
+ o.Name = k
+ o.Scheme = url.Scheme
+ o.Host = url.Host
+ o.PathPrefix = url.Path
+ o.Timeout = time.Duration(o.TimeoutSecs) * time.Second
+ o.BackfillTolerance = time.Duration(o.BackfillToleranceSecs) * time.Second
+ o.TimeseriesRetention = time.Duration(o.TimeseriesRetentionFactor)
+ o.TimeseriesTTL = time.Duration(o.TimeseriesTTLSecs) * time.Second
+ o.FastForwardTTL = time.Duration(o.FastForwardTTLSecs) * time.Second
+ o.MaxTTL = time.Duration(o.MaxTTLSecs) * time.Second
+
+ if o.CompressableTypeList != nil {
+ o.CompressableTypes = make(map[string]bool)
+ for _, v := range o.CompressableTypeList {
+ o.CompressableTypes[v] = true
+ }
+ }
+
+ if o.CacheKeyPrefix == "" {
+ o.CacheKeyPrefix = o.Host
+ }
+
+ nc, ok := NegativeCacheConfigs[o.NegativeCacheName]
+ if !ok {
+ return fmt.Errorf(`invalid negative cache name: %s`, o.NegativeCacheName)
+ }
+
+ nc2 := map[int]time.Duration{}
+ for c, s := range nc {
+ ci, _ := strconv.Atoi(c)
+ nc2[ci] = time.Duration(s) * time.Second
+ }
+ o.NegativeCache = nc2
+
+ // enforce MaxTTL
+ if o.TimeseriesTTLSecs > o.MaxTTLSecs {
+ o.TimeseriesTTLSecs = o.MaxTTLSecs
+ o.TimeseriesTTL = o.MaxTTL
+ }
+
+ // unlikely but why not spend a few nanoseconds to check it at startup
+ if o.FastForwardTTLSecs > o.MaxTTLSecs {
+ o.FastForwardTTLSecs = o.MaxTTLSecs
+ o.FastForwardTTL = o.MaxTTL
+ }
+
+ Origins[k] = o
+ }
+
+ for _, c := range Caches {
+ c.Index.FlushInterval = time.Duration(c.Index.FlushIntervalSecs) * time.Second
+ c.Index.ReapInterval = time.Duration(c.Index.ReapIntervalSecs) * time.Second
+ }
+
+ return nil
+}
diff --git a/internal/config/loader_test.go b/internal/config/loader_test.go
new file mode 100644
index 000000000..8be0fdfb2
--- /dev/null
+++ b/internal/config/loader_test.go
@@ -0,0 +1,668 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "fmt"
+ "strconv"
+ "testing"
+ "time"
+)
+
+func TestLoadConfiguration(t *testing.T) {
+ a := []string{"-origin-type", "testing", "-origin-url", "http://prometheus:9090/test/path"}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if Origins["default"].TimeseriesRetention != 1024 {
+ t.Errorf("expected 1024, got %d", Origins["default"].TimeseriesRetention)
+ }
+
+ if Origins["default"].FastForwardTTL != time.Duration(15)*time.Second {
+ t.Errorf("expected 15, got %s", Origins["default"].FastForwardTTL)
+ }
+
+ if Caches["default"].Index.ReapInterval != time.Duration(3)*time.Second {
+ t.Errorf("expected 3, got %s", Caches["default"].Index.ReapInterval)
+ }
+
+}
+
+func TestLoadConfigurationFileFailures(t *testing.T) {
+
+ tests := []struct {
+ filename string
+ expected string
+ }{
+ { // Case 0
+ "../../testdata/test.missing-origin-url.conf",
+ `missing origin-url for origin "2"`,
+ },
+ { // Case 1
+ "../../testdata/test.bad_origin_url.conf",
+ fmt.Sprintf(`parse %s: first path segment in URL cannot contain colon`, "sasdf_asd[as;://asdf923_-=a*"),
+ },
+ { // Case 2
+ "../../testdata/test.missing_origin_type.conf",
+ `missing origin-type for origin "test"`,
+ },
+ { // Case 3
+ "../../testdata/test.bad-cache-name.conf",
+ `invalid cache name [test_fail] provided in origin config [test]`,
+ },
+ { // Case 4
+ "../../testdata/test.invalid-negative-cache-1.conf",
+ `invalid negative cache config in default: a is not a valid status code`,
+ },
+ { // Case 5
+ "../../testdata/test.invalid-negative-cache-2.conf",
+ `invalid negative cache config in default: 1212 is not a valid status code`,
+ },
+ { // Case 6
+ "../../testdata/test.invalid-negative-cache-3.conf",
+ `invalid negative cache name: foo`,
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ err := Load("trickster-test", "0", []string{"-config", test.filename})
+ if err == nil {
+ t.Errorf("expected error `%s` got nothing", test.expected)
+ } else if err.Error() != test.expected {
+ t.Errorf("expected error `%s` got `%s`", test.expected, err.Error())
+ }
+
+ })
+ }
+
+}
+
+func TestLoadConfigurationMissingOriginURL(t *testing.T) {
+ expected := `no valid origins configured`
+ a := []string{"-origin-type", "testing"}
+ err := Load("trickster-test", "0", a)
+ if err == nil {
+ t.Errorf("expected error `%s` got nothing", expected)
+ } else if err.Error() != expected {
+ t.Errorf("expected error `%s` got `%s`", expected, err.Error())
+ }
+}
+
+func TestFullLoadConfiguration(t *testing.T) {
+ a := []string{"-config", "../../testdata/test.full.conf"}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // Test Proxy Server
+ if Frontend.ListenPort != 57821 {
+ t.Errorf("expected 57821, got %d", Frontend.ListenPort)
+ }
+
+ if Frontend.ListenAddress != "test" {
+ t.Errorf("expected test, got %s", Frontend.ListenAddress)
+ }
+
+ if Frontend.TLSListenAddress != "test-tls" {
+ t.Errorf("expected test-tls, got %s", Frontend.TLSListenAddress)
+ }
+
+ if Frontend.TLSListenPort != 38821 {
+ t.Errorf("expected 38821, got %d", Frontend.TLSListenPort)
+ }
+
+ // Test Metrics Server
+ if Metrics.ListenPort != 57822 {
+ t.Errorf("expected 57821, got %d", Metrics.ListenPort)
+ }
+
+ if Metrics.ListenAddress != "metrics_test" {
+ t.Errorf("expected test, got %s", Metrics.ListenAddress)
+ }
+
+ // Test Logging
+ if Logging.LogLevel != "test_log_level" {
+ t.Errorf("expected test_log_level, got %s", Logging.LogLevel)
+ }
+
+ if Logging.LogFile != "test_file" {
+ t.Errorf("expected test_file, got %s", Logging.LogFile)
+ }
+
+ // Test Origins
+
+ o, ok := Origins["test"]
+ if !ok {
+ t.Errorf("unable to find origin config: %s", "test")
+ return
+ }
+
+ if o.OriginType != "test_type" {
+ t.Errorf("expected test_type, got %s", o.OriginType)
+ }
+
+ if o.CacheName != "test" {
+ t.Errorf("expected test, got %s", o.CacheName)
+ }
+
+ if o.Scheme != "scheme" {
+ t.Errorf("expected scheme, got %s", o.Scheme)
+ }
+
+ if o.Host != "test_host" {
+ t.Errorf("expected test_host, got %s", o.Host)
+ }
+
+ if o.PathPrefix != "/test_path_prefix" {
+ t.Errorf("expected test_path_prefix, got %s", o.PathPrefix)
+ }
+
+ if o.TimeseriesRetentionFactor != 666 {
+ t.Errorf("expected 666, got %d", o.TimeseriesRetentionFactor)
+ }
+
+ if o.TimeseriesEvictionMethod != EvictionMethodLRU {
+ t.Errorf("expected %s, got %s", EvictionMethodLRU, o.TimeseriesEvictionMethod)
+ }
+
+ if !o.FastForwardDisable {
+ t.Errorf("expected fast_forward_disable true, got %t", o.FastForwardDisable)
+ }
+
+ if o.BackfillToleranceSecs != 301 {
+ t.Errorf("expected 301, got %d", o.BackfillToleranceSecs)
+ }
+
+ if o.TimeoutSecs != 37 {
+ t.Errorf("expected 37, got %d", o.TimeoutSecs)
+ }
+
+ if o.IsDefault != true {
+ t.Errorf("expected true got %t", o.IsDefault)
+ }
+
+ if o.MaxIdleConns != 23 {
+ t.Errorf("expected %d got %d", 23, o.MaxIdleConns)
+ }
+
+ if o.KeepAliveTimeoutSecs != 7 {
+ t.Errorf("expected %d got %d", 7, o.KeepAliveTimeoutSecs)
+ }
+
+ // MaxTTLSecs is 300, thus should override TimeseriesTTLSecs = 8666
+ if o.TimeseriesTTLSecs != 300 {
+ t.Errorf("expected 300, got %d", o.TimeseriesTTLSecs)
+ }
+
+ // MaxTTLSecs is 300, thus should override FastForwardTTLSecs = 382
+ if o.FastForwardTTLSecs != 300 {
+ t.Errorf("expected 300, got %d", o.FastForwardTTLSecs)
+ }
+
+ if o.TLS == nil {
+ t.Errorf("expected tls config for origin %s, got nil", "test")
+ }
+
+ if !o.TLS.InsecureSkipVerify {
+ t.Errorf("expected true got %t", o.TLS.InsecureSkipVerify)
+ }
+
+ if o.TLS.FullChainCertPath != "../../testdata/test.01.cert.pem" {
+ t.Errorf("expected ../../testdata/test.01.cert.pem got %s", o.TLS.FullChainCertPath)
+ }
+
+ if o.TLS.PrivateKeyPath != "../../testdata/test.01.key.pem" {
+ t.Errorf("expected ../../testdata/test.01.key.pem got %s", o.TLS.PrivateKeyPath)
+ }
+
+ if o.TLS.ClientCertPath != "test_client_cert" {
+ t.Errorf("expected test_client_cert got %s", o.TLS.ClientCertPath)
+ }
+
+ if o.TLS.ClientKeyPath != "test_client_key" {
+ t.Errorf("expected test_client_key got %s", o.TLS.ClientKeyPath)
+ }
+
+ // Test Caches
+
+ c, ok := Caches["test"]
+ if !ok {
+ t.Errorf("unable to find cache config: %s", "test")
+ return
+ }
+
+ if c.CacheType != "redis" {
+ t.Errorf("expected redis, got %s", c.CacheType)
+ }
+
+ if c.Index.ReapIntervalSecs != 4 {
+ t.Errorf("expected 4, got %d", c.Index.ReapIntervalSecs)
+ }
+
+ if c.Index.FlushIntervalSecs != 6 {
+ t.Errorf("expected 6, got %d", c.Index.FlushIntervalSecs)
+ }
+
+ if c.Index.MaxSizeBytes != 536870913 {
+ t.Errorf("expected 536870913, got %d", c.Index.MaxSizeBytes)
+ }
+
+ if c.Index.MaxSizeBackoffBytes != 16777217 {
+ t.Errorf("expected 16777217, got %d", c.Index.MaxSizeBackoffBytes)
+ }
+
+ if c.Index.MaxSizeObjects != 80 {
+ t.Errorf("expected 80, got %d", c.Index.MaxSizeObjects)
+ }
+
+ if c.Index.MaxSizeBackoffObjects != 20 {
+ t.Errorf("expected 20, got %d", c.Index.MaxSizeBackoffObjects)
+ }
+
+ if c.Index.ReapIntervalSecs != 4 {
+ t.Errorf("expected 4, got %d", c.Index.ReapIntervalSecs)
+ }
+
+ if c.Redis.ClientType != "test_redis_type" {
+ t.Errorf("expected test_redis_type, got %s", c.Redis.ClientType)
+ }
+
+ if c.Redis.Protocol != "test_protocol" {
+ t.Errorf("expected test_protocol, got %s", c.Redis.Protocol)
+ }
+
+ if c.Redis.Endpoint != "test_endpoint" {
+ t.Errorf("expected test_endpoint, got %s", c.Redis.Endpoint)
+ }
+
+ if c.Redis.SentinelMaster != "test_master" {
+ t.Errorf("expected test_master, got %s", c.Redis.SentinelMaster)
+ }
+
+ if c.Redis.Password != "test_password" {
+ t.Errorf("expected test_password, got %s", c.Redis.Password)
+ }
+
+ if c.Redis.DB != 42 {
+ t.Errorf("expected 42, got %d", c.Redis.DB)
+ }
+
+ if c.Redis.MaxRetries != 6 {
+ t.Errorf("expected 6, got %d", c.Redis.MaxRetries)
+ }
+
+ if c.Redis.MinRetryBackoffMS != 9 {
+ t.Errorf("expected 9, got %d", c.Redis.MinRetryBackoffMS)
+ }
+
+ if c.Redis.MaxRetryBackoffMS != 513 {
+ t.Errorf("expected 513, got %d", c.Redis.MaxRetryBackoffMS)
+ }
+
+ if c.Redis.DialTimeoutMS != 5001 {
+ t.Errorf("expected 5001, got %d", c.Redis.DialTimeoutMS)
+ }
+
+ if c.Redis.ReadTimeoutMS != 3001 {
+ t.Errorf("expected 3001, got %d", c.Redis.ReadTimeoutMS)
+ }
+
+ if c.Redis.WriteTimeoutMS != 3002 {
+ t.Errorf("expected 3002, got %d", c.Redis.WriteTimeoutMS)
+ }
+
+ if c.Redis.PoolSize != 21 {
+ t.Errorf("expected 21, got %d", c.Redis.PoolSize)
+ }
+
+ if c.Redis.MinIdleConns != 5 {
+ t.Errorf("expected 5, got %d", c.Redis.PoolSize)
+ }
+
+ if c.Redis.MaxConnAgeMS != 2000 {
+ t.Errorf("expected 2000, got %d", c.Redis.MaxConnAgeMS)
+ }
+
+ if c.Redis.PoolTimeoutMS != 4001 {
+ t.Errorf("expected 4001, got %d", c.Redis.PoolTimeoutMS)
+ }
+
+ if c.Redis.IdleTimeoutMS != 300001 {
+ t.Errorf("expected 300001, got %d", c.Redis.IdleTimeoutMS)
+ }
+
+ if c.Redis.IdleCheckFrequencyMS != 60001 {
+ t.Errorf("expected 60001, got %d", c.Redis.IdleCheckFrequencyMS)
+ }
+
+ if c.Filesystem.CachePath != "test_cache_path" {
+ t.Errorf("expected test_cache_path, got %s", c.Filesystem.CachePath)
+ }
+
+ if c.BBolt.Filename != "test_filename" {
+ t.Errorf("expected test_filename, got %s", c.BBolt.Filename)
+ }
+
+ if c.BBolt.Bucket != "test_bucket" {
+ t.Errorf("expected test_bucket, got %s", c.BBolt.Bucket)
+ }
+
+ if c.Badger.Directory != "test_directory" {
+ t.Errorf("expected test_directory, got %s", c.Badger.Directory)
+ }
+
+ if c.Badger.ValueDirectory != "test_value_directory" {
+ t.Errorf("expected test_value_directory, got %s", c.Badger.ValueDirectory)
+ }
+}
+
+func TestEmptyLoadConfiguration(t *testing.T) {
+ a := []string{"-config", "../../testdata/test.empty.conf"}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if len(Origins) != 1 {
+ // we define a "test" cache, but never reference it by an origin,
+ // so it should not make it into the running config
+ t.Errorf("expected %d, got %d", 1, len(Origins))
+ }
+
+ // Test Proxy Server
+ if Frontend.ListenPort != defaultProxyListenPort {
+ t.Errorf("expected %d, got %d", defaultProxyListenPort, Frontend.ListenPort)
+ }
+
+ if Frontend.ListenAddress != defaultProxyListenAddress {
+ t.Errorf("expected '%s', got '%s'", defaultProxyListenAddress, Frontend.ListenAddress)
+ }
+
+ // Test Metrics Server
+ if Metrics.ListenPort != defaultMetricsListenPort {
+ t.Errorf("expected %d, got %d", defaultMetricsListenPort, Metrics.ListenPort)
+ }
+
+ if Metrics.ListenAddress != defaultMetricsListenAddress {
+ t.Errorf("expected '%s', got '%s'", defaultMetricsListenAddress, Metrics.ListenAddress)
+ }
+
+ // Test Logging
+ if Logging.LogLevel != defaultLogLevel {
+ t.Errorf("expected %s, got %s", defaultLogLevel, Logging.LogLevel)
+ }
+
+ if Logging.LogFile != defaultLogFile {
+ t.Errorf("expected '%s', got '%s'", defaultLogFile, Logging.LogFile)
+ }
+
+ // Test Origins
+
+ o, ok := Origins["test"]
+ if !ok {
+ t.Errorf("unable to find origin config: %s", "test")
+ return
+ }
+
+ if o.OriginType != "test" {
+ t.Errorf("expected %s origin type, got %s", "test", o.OriginType)
+ }
+
+ if o.CacheName != defaultOriginCacheName {
+ t.Errorf("expected %s, got %s", defaultOriginCacheName, o.CacheName)
+ }
+
+ if o.Scheme != "http" {
+ t.Errorf("expected %s, got %s", "http", o.Scheme)
+ }
+
+ if o.Host != "1" {
+ t.Errorf("expected %s, got %s", "1", o.Host)
+ }
+
+ if o.PathPrefix != "" {
+ t.Errorf("expected '%s', got '%s'", "", o.PathPrefix)
+ }
+
+ if o.TimeseriesRetentionFactor != defaultOriginTRF {
+ t.Errorf("expected %d, got %d", defaultOriginTRF, o.TimeseriesRetentionFactor)
+ }
+
+ if o.FastForwardDisable {
+ t.Errorf("expected fast_forward_disable false, got %t", o.FastForwardDisable)
+ }
+
+ if o.BackfillToleranceSecs != defaultBackfillToleranceSecs {
+ t.Errorf("expected %d, got %d", defaultBackfillToleranceSecs, o.BackfillToleranceSecs)
+ }
+
+ if o.TimeoutSecs != defaultOriginTimeoutSecs {
+ t.Errorf("expected %d, got %d", defaultOriginTimeoutSecs, o.TimeoutSecs)
+ }
+
+ if o.TimeseriesTTLSecs != defaultTimeseriesTTLSecs {
+ t.Errorf("expected %d, got %d", defaultTimeseriesTTLSecs, o.TimeseriesTTLSecs)
+ }
+
+ if o.FastForwardTTLSecs != defaultFastForwardTTLSecs {
+ t.Errorf("expected %d, got %d", defaultFastForwardTTLSecs, o.FastForwardTTLSecs)
+ }
+
+ c, ok := Caches["default"]
+ if !ok {
+ t.Errorf("unable to find cache config: %s", "default")
+ return
+ }
+
+ if c.CacheType != defaultCacheType {
+ t.Errorf("expected %s, got %s", defaultCacheType, c.CacheType)
+ }
+
+ if c.Index.ReapIntervalSecs != defaultCacheIndexReap {
+ t.Errorf("expected %d, got %d", defaultCacheIndexReap, c.Index.ReapIntervalSecs)
+ }
+
+ if c.Index.FlushIntervalSecs != defaultCacheIndexFlush {
+ t.Errorf("expected %d, got %d", defaultCacheIndexFlush, c.Index.FlushIntervalSecs)
+ }
+
+ if c.Index.MaxSizeBytes != defaultCacheMaxSizeBytes {
+ t.Errorf("expected %d, got %d", defaultCacheMaxSizeBytes, c.Index.MaxSizeBytes)
+ }
+
+ if c.Index.MaxSizeBackoffBytes != defaultMaxSizeBackoffBytes {
+ t.Errorf("expected %d, got %d", defaultMaxSizeBackoffBytes, c.Index.MaxSizeBackoffBytes)
+ }
+
+ if c.Index.MaxSizeObjects != defaultMaxSizeObjects {
+ t.Errorf("expected %d, got %d", defaultMaxSizeObjects, c.Index.MaxSizeObjects)
+ }
+
+ if c.Index.MaxSizeBackoffObjects != defaultMaxSizeBackoffObjects {
+ t.Errorf("expected %d, got %d", defaultMaxSizeBackoffObjects, c.Index.MaxSizeBackoffObjects)
+ }
+
+ if c.Index.ReapIntervalSecs != 3 {
+ t.Errorf("expected 3, got %d", c.Index.ReapIntervalSecs)
+ }
+
+ if c.Redis.ClientType != defaultRedisClientType {
+ t.Errorf("expected %s, got %s", defaultRedisClientType, c.Redis.ClientType)
+ }
+
+ if c.Redis.Protocol != defaultRedisProtocol {
+ t.Errorf("expected %s, got %s", defaultRedisProtocol, c.Redis.Protocol)
+ }
+
+ if c.Redis.Endpoint != "redis:6379" {
+ t.Errorf("expected redis:6379, got %s", c.Redis.Endpoint)
+ }
+
+ if c.Redis.SentinelMaster != "" {
+ t.Errorf("expected '', got %s", c.Redis.SentinelMaster)
+ }
+
+ if c.Redis.Password != "" {
+ t.Errorf("expected '', got %s", c.Redis.Password)
+ }
+
+ if c.Redis.DB != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.DB)
+ }
+
+ if c.Redis.MaxRetries != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.MaxRetries)
+ }
+
+ if c.Redis.MinRetryBackoffMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.MinRetryBackoffMS)
+ }
+
+ if c.Redis.MaxRetryBackoffMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.MaxRetryBackoffMS)
+ }
+
+ if c.Redis.DialTimeoutMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.DialTimeoutMS)
+ }
+
+ if c.Redis.ReadTimeoutMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.ReadTimeoutMS)
+ }
+
+ if c.Redis.WriteTimeoutMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.WriteTimeoutMS)
+ }
+
+ if c.Redis.PoolSize != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.PoolSize)
+ }
+
+ if c.Redis.MinIdleConns != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.PoolSize)
+ }
+
+ if c.Redis.MaxConnAgeMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.MaxConnAgeMS)
+ }
+
+ if c.Redis.PoolTimeoutMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.PoolTimeoutMS)
+ }
+
+ if c.Redis.IdleTimeoutMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.IdleTimeoutMS)
+ }
+
+ if c.Redis.IdleCheckFrequencyMS != 0 {
+ t.Errorf("expected 0, got %d", c.Redis.IdleCheckFrequencyMS)
+ }
+
+ if c.Filesystem.CachePath != "/tmp/trickster" {
+ t.Errorf("expected /tmp/trickster, got %s", c.Filesystem.CachePath)
+ }
+
+ if c.BBolt.Filename != "trickster.db" {
+ t.Errorf("expected trickster.db, got %s", c.BBolt.Filename)
+ }
+
+ if c.BBolt.Bucket != "trickster" {
+ t.Errorf("expected trickster, got %s", c.BBolt.Bucket)
+ }
+
+ if c.Badger.Directory != "/tmp/trickster" {
+ t.Errorf("expected /tmp/trickster, got %s", c.Badger.Directory)
+ }
+
+ if c.Badger.ValueDirectory != "/tmp/trickster" {
+ t.Errorf("expected /tmp/trickster, got %s", c.Badger.ValueDirectory)
+ }
+}
+
+func TestLoadConfigurationVersion(t *testing.T) {
+ a := []string{"-version"}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if !Flags.PrintVersion {
+ t.Errorf("expected true got false")
+ }
+}
+
+func TestLoadConfigurationBadPath(t *testing.T) {
+
+ const badPath = "/afeas/aasdvasvasdf48/ag4a4gas"
+
+ a := []string{"-config", badPath}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err == nil {
+ t.Errorf("expected error: open %s: no such file or directory", badPath)
+ }
+}
+
+func TestLoadConfigurationBadUrl(t *testing.T) {
+ const badURL = ":httap:]/]/example.com9091"
+ a := []string{"-origin-url", badURL}
+ err := Load("trickster-test", "0", a)
+ if err == nil {
+ t.Errorf("expected error: parse %s: missing protocol scheme", badURL)
+ }
+}
+
+func TestLoadConfigurationWarning1(t *testing.T) {
+
+ a := []string{"-config", "../../testdata/test.warning1.conf"}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ expected := 1
+ l := len(LoaderWarnings)
+
+ if l != expected {
+ t.Errorf("exepcted %d got %d", expected, l)
+ }
+
+}
+
+func TestLoadConfigurationWarning2(t *testing.T) {
+
+ a := []string{"-config", "../../testdata/test.warning2.conf"}
+ // it should not error if config path is not set
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+ expected := 1
+ l := len(LoaderWarnings)
+
+ if l != expected {
+ t.Errorf("exepcted %d got %d", expected, l)
+ }
+
+}
diff --git a/internal/config/origin_types.go b/internal/config/origin_types.go
new file mode 100644
index 000000000..3cfdeda08
--- /dev/null
+++ b/internal/config/origin_types.go
@@ -0,0 +1,62 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import "strconv"
+
+// OriginType enumerates the supported origin types
+type OriginType int
+
+const (
+ // OriginTypeRPC represents thee Reverse Proxy Cache origin type
+ OriginTypeRPC = OriginType(iota)
+ // OriginTypePrometheus represents the Prometheus origin type
+ OriginTypePrometheus
+ // OriginTypeInfluxDB represents the InfluxDB origin type
+ OriginTypeInfluxDB
+ // OriginTypeIronDB represents the IRONdb origin type
+ OriginTypeIronDB
+ // OriginTypeClickHouse represents the ClickHouse origin type
+ OriginTypeClickHouse
+)
+
+var originTypeNames = map[string]OriginType{
+ "rpc": OriginTypeRPC,
+ "reverseproxycache": OriginTypeRPC,
+ "prometheus": OriginTypePrometheus,
+ "influxdb": OriginTypeInfluxDB,
+ "irondb": OriginTypeIronDB,
+ "clickhouse": OriginTypeClickHouse,
+}
+
+var originTypeValues = map[OriginType]string{
+ OriginTypeRPC: "rpc",
+ OriginTypePrometheus: "prometheus",
+ OriginTypeInfluxDB: "influxdb",
+ OriginTypeIronDB: "irondb",
+ OriginTypeClickHouse: "clickhouse",
+}
+
+func (t OriginType) String() string {
+ if v, ok := originTypeValues[t]; ok {
+ return v
+ }
+ return strconv.Itoa(int(t))
+}
+
+// IsValidOriginType returns true if the provided OriginType is valid for use with Trickster
+func IsValidOriginType(t string) bool {
+ _, ok := originTypeNames[t]
+ return ok
+}
diff --git a/internal/config/origin_types_test.go b/internal/config/origin_types_test.go
new file mode 100644
index 000000000..028aac7f6
--- /dev/null
+++ b/internal/config/origin_types_test.go
@@ -0,0 +1,64 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "strconv"
+ "testing"
+)
+
+func TestOriginTypeString(t *testing.T) {
+
+ t1 := OriginTypeRPC
+ t2 := OriginTypePrometheus
+ var t3 OriginType = 13
+
+ if t1.String() != "rpc" {
+ t.Errorf("expected %s got %s", "rpc", t1.String())
+ }
+
+ if t2.String() != "prometheus" {
+ t.Errorf("expected %s got %s", "prometheus", t2.String())
+ }
+
+ if t3.String() != "13" {
+ t.Errorf("expected %s got %s", "13", t3.String())
+ }
+
+}
+
+func TestIsValidOriginType(t *testing.T) {
+
+ tests := []struct {
+ o string
+ expected bool
+ }{
+ {"rpc", true},
+ {"prometheus", true},
+ {"", false},
+ {"invalid", false},
+ {"influxdb", true},
+ {"irondb", true},
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ res := IsValidOriginType(test.o)
+ if test.expected != res {
+ t.Errorf("expected %t got %t", test.expected, res)
+ }
+ })
+ }
+
+}
diff --git a/internal/config/path.go b/internal/config/path.go
new file mode 100644
index 000000000..1c5c7280f
--- /dev/null
+++ b/internal/config/path.go
@@ -0,0 +1,209 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "io"
+ "net/http"
+ "net/url"
+ "strconv"
+
+ "github.com/Comcast/trickster/internal/proxy/methods"
+ ts "github.com/Comcast/trickster/internal/util/strings"
+)
+
+// PathMatchType enumerates the types of Path Matches used when registering Paths with the Router
+type PathMatchType int
+
+// KeyHasherFunc is a custom function that returns a hashed key value string for cache objects
+type KeyHasherFunc func(path string, params url.Values, headers http.Header, body io.ReadCloser, extra string) string
+
+const (
+ // PathMatchTypeExact indicates the router will map the Path by exact match against incoming requests
+ PathMatchTypeExact = PathMatchType(iota)
+ // PathMatchTypePrefix indicates the router will map the Path by prefix against incoming requests
+ PathMatchTypePrefix
+)
+
+var pathMatchTypeNames = map[string]PathMatchType{
+ "exact": PathMatchTypeExact,
+ "prefix": PathMatchTypePrefix,
+}
+
+var pathMatchTypeValues = map[PathMatchType]string{
+ PathMatchTypeExact: "exact",
+ PathMatchTypePrefix: "prefix",
+}
+
+func (t PathMatchType) String() string {
+ if v, ok := pathMatchTypeValues[t]; ok {
+ return v
+ }
+ return strconv.Itoa(int(t))
+}
+
+// PathConfig defines a URL Path that is associated with an HTTP Handler
+type PathConfig struct {
+ // Path indicates the HTTP Request's URL PATH to which this configuration applies
+ Path string `toml:"path"`
+ // MatchTypeName indicates the type of path match the router will apply to the path ('exact' or 'prefix')
+ MatchTypeName string `toml:"match_type"`
+ // HandlerName provides the name of the HTTP handler to use
+ HandlerName string `toml:"handler"`
+ // Methods provides the list of permitted HTTP request methods for this Path
+ Methods []string `toml:"methods"`
+ // CacheKeyParams provides the list of http request query parameters to be included in the hash for each request's cache key
+ CacheKeyParams []string `toml:"cache_key_params"`
+ // CacheKeyHeaders provides the list of http request headers to be included in the hash for each request's cache key
+ CacheKeyHeaders []string `toml:"cache_key_headers"`
+ // CacheKeyFormFields provides the list of http request body fields to be included in the hash for each request's cache key
+ CacheKeyFormFields []string `toml:"cache_key_form_fields"`
+ // RequestHeaders is a map of headers that will be added to requests to the upstream Origin for this path
+ RequestHeaders map[string]string `toml:"request_headers"`
+ // RequestParams is a map of headers that will be added to requests to the upstream Origin for this path
+ RequestParams map[string]string `toml:"request_params"`
+ // ResponseHeaders is a map of http headers that will be added to responses to the downstream client
+ ResponseHeaders map[string]string `toml:"response_headers"`
+ // ResponseCode sets a custom response code to be sent to downstream clients for this path.
+ ResponseCode int `toml:"response_code"`
+ // ResponseBody sets a custom response body to be sent to the donstream client for this path.
+ ResponseBody string `toml:"response_body"`
+ // NoMetrics, when set to true, disables metrics decoration for the path
+ NoMetrics bool `toml:"no_metrics"`
+ // CollapsedForwardingName indicates 'basic' or 'progressive' Collapsed Forwarding to be used by this path.
+ CollapsedForwardingName string `toml:"collapsed_forwarding"`
+
+ // Synthesized PathConfig Values
+ //
+ // Handler is the HTTP Handler represented by the Path's HandlerName
+ Handler http.Handler `toml:"-"`
+ // HasCustomResponseBody is a boolean indicating if the response body is custom
+ // this flag allows an empty string response to be configured as a return value
+ HasCustomResponseBody bool `toml:"-"`
+ // ResponseBodyBytes provides a byte slice version of the ResponseBody value
+ ResponseBodyBytes []byte `toml:"-"`
+ // MatchType is the PathMatchType representation of MatchTypeName
+ MatchType PathMatchType `toml:"-"`
+ // CollapsedForwardingType is the typed representation of CollapsedForwardingName
+ CollapsedForwardingType CollapsedForwardingType `toml:"-"`
+ // OriginConfig is the reference to the PathConfig's parent Origin Config
+ OriginConfig *OriginConfig `toml:"-"`
+ // KeyHasher points to an optional function that hashes the cacheKey with a custom algorithm
+ // NOTE: This is used by some origins like IronDB, but is not configurable by end users
+ // due to a bug in the vendored toml package, this must be a slice to avoid panic
+ KeyHasher []KeyHasherFunc `toml:"-"`
+
+ custom []string `toml:"-"`
+}
+
+// NewPathConfig returns a newly-instantiated *PathConfig
+func NewPathConfig() *PathConfig {
+ return &PathConfig{
+ Path: "/",
+ Methods: methods.CacheableHTTPMethods(),
+ HandlerName: "proxy",
+ MatchTypeName: "exact",
+ MatchType: PathMatchTypeExact,
+ CollapsedForwardingName: "basic",
+ CollapsedForwardingType: CFTypeBasic,
+ CacheKeyParams: make([]string, 0),
+ CacheKeyHeaders: make([]string, 0),
+ CacheKeyFormFields: make([]string, 0),
+ custom: make([]string, 0),
+ RequestHeaders: make(map[string]string),
+ RequestParams: make(map[string]string),
+ ResponseHeaders: make(map[string]string),
+ KeyHasher: nil,
+ }
+}
+
+// Clone returns an exact copy of the subject PathConfig
+func (p *PathConfig) Clone() *PathConfig {
+ c := &PathConfig{
+ Path: p.Path,
+ OriginConfig: p.OriginConfig,
+ MatchTypeName: p.MatchTypeName,
+ MatchType: p.MatchType,
+ HandlerName: p.HandlerName,
+ Handler: p.Handler,
+ RequestHeaders: ts.CloneMap(p.RequestHeaders),
+ RequestParams: ts.CloneMap(p.RequestParams),
+ ResponseHeaders: ts.CloneMap(p.ResponseHeaders),
+ ResponseBody: p.ResponseBody,
+ ResponseBodyBytes: p.ResponseBodyBytes,
+ CollapsedForwardingName: p.CollapsedForwardingName,
+ CollapsedForwardingType: p.CollapsedForwardingType,
+ NoMetrics: p.NoMetrics,
+ HasCustomResponseBody: p.HasCustomResponseBody,
+ Methods: make([]string, len(p.Methods)),
+ CacheKeyParams: make([]string, len(p.CacheKeyParams)),
+ CacheKeyHeaders: make([]string, len(p.CacheKeyHeaders)),
+ CacheKeyFormFields: make([]string, len(p.CacheKeyFormFields)),
+ custom: make([]string, len(p.custom)),
+ KeyHasher: p.KeyHasher,
+ }
+ copy(c.Methods, p.Methods)
+ copy(c.CacheKeyParams, p.CacheKeyParams)
+ copy(c.CacheKeyHeaders, p.CacheKeyHeaders)
+ copy(c.CacheKeyFormFields, p.CacheKeyFormFields)
+ copy(c.custom, p.custom)
+ return c
+
+}
+
+// Merge merges the non-default values of the provided PathConfig into the subject PathConfig
+func (p *PathConfig) Merge(p2 *PathConfig) {
+
+ if p2.OriginConfig != nil {
+ p.OriginConfig = p2.OriginConfig
+ }
+
+ for _, c := range p2.custom {
+ switch c {
+ case "path":
+ p.Path = p2.Path
+ case "match_type":
+ p.MatchType = p2.MatchType
+ p.MatchTypeName = p2.MatchTypeName
+ case "handler":
+ p.HandlerName = p2.HandlerName
+ p.Handler = p2.Handler
+ case "methods":
+ p.Methods = p2.Methods
+ case "cache_key_params":
+ p.CacheKeyParams = p2.CacheKeyParams
+ case "cache_key_headers":
+ p.CacheKeyHeaders = p2.CacheKeyHeaders
+ case "cache_key_form_fields":
+ p.CacheKeyFormFields = p2.CacheKeyFormFields
+ case "request_headers":
+ p.RequestHeaders = p2.RequestHeaders
+ case "request_params":
+ p.RequestParams = p2.RequestParams
+ case "response_headers":
+ p.ResponseHeaders = p2.ResponseHeaders
+ case "response_code":
+ p.ResponseCode = p2.ResponseCode
+ case "response_body":
+ p.ResponseBody = p2.ResponseBody
+ p.HasCustomResponseBody = true
+ p.ResponseBodyBytes = p2.ResponseBodyBytes
+ case "no_metrics":
+ p.NoMetrics = p2.NoMetrics
+ case "collapsed_forwarding":
+ p.CollapsedForwardingName = p2.CollapsedForwardingName
+ p.CollapsedForwardingType = p2.CollapsedForwardingType
+ }
+ }
+}
diff --git a/internal/config/path_test.go b/internal/config/path_test.go
new file mode 100644
index 000000000..25c9ac970
--- /dev/null
+++ b/internal/config/path_test.go
@@ -0,0 +1,162 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "net/http"
+ "testing"
+)
+
+func TestPMTString(t *testing.T) {
+
+ t1 := PathMatchTypeExact
+ t2 := PathMatchTypePrefix
+
+ var t3 PathMatchType = 3
+
+ if t1.String() != "exact" {
+ t.Errorf("expected %s got %s", "exact", t1.String())
+ }
+
+ if t2.String() != "prefix" {
+ t.Errorf("expected %s got %s", "prefix", t2.String())
+ }
+
+ if t3.String() != "3" {
+ t.Errorf("expected %s got %s", "3", t3.String())
+ }
+
+}
+
+func TestNewPathConfig(t *testing.T) {
+
+ pc := NewPathConfig()
+
+ if pc == nil {
+ t.Errorf("expected non-nil value you for %s", "PathConfig")
+ }
+
+ if pc.HandlerName != "proxy" {
+ t.Errorf("expected value %s, got %s", "proxy", pc.HandlerName)
+ }
+
+}
+
+func TestPathClone(t *testing.T) {
+
+ pc := NewPathConfig()
+ pc2 := pc.Clone()
+
+ if pc2 == nil {
+ t.Errorf("expected non-nil value you for %s", "PathConfig")
+ }
+
+ if pc2.HandlerName != "proxy" {
+ t.Errorf("expected value %s, got %s", "proxy", pc2.HandlerName)
+ }
+
+}
+
+func TestPathMerge(t *testing.T) {
+
+ pc := NewPathConfig()
+ pc2 := pc.Clone()
+
+ pc2.OriginConfig = NewOriginConfig()
+
+ pc2.custom = []string{"path", "match_type", "handler", "methods", "cache_key_params", "cache_key_headers", "cache_key_form_fields",
+ "request_headers", "request_params", "response_headers", "response_code", "response_body", "no_metrics", "collapsed_forwarding"}
+
+ expectedPath := "testPath"
+ expectedHandlerName := "testHandler"
+
+ pc2.Path = expectedPath
+ pc2.MatchType = PathMatchTypePrefix
+ pc2.HandlerName = expectedHandlerName
+ pc2.Methods = []string{http.MethodPost}
+ pc2.CacheKeyParams = []string{"params"}
+ pc2.CacheKeyHeaders = []string{"headers"}
+ pc2.CacheKeyFormFields = []string{"fields"}
+ pc2.RequestHeaders = map[string]string{"header1": "1"}
+ pc2.RequestParams = map[string]string{"param1": "foo"}
+ pc2.ResponseHeaders = map[string]string{"header2": "2"}
+ pc2.ResponseCode = 404
+ pc2.ResponseBody = "trickster"
+ pc2.NoMetrics = true
+ pc2.CollapsedForwardingName = "progressive"
+ pc2.CollapsedForwardingType = CFTypeProgressive
+
+ pc.Merge(pc2)
+
+ if pc.Path != expectedPath {
+ t.Errorf("expected %s got %s", expectedPath, pc.Path)
+ }
+
+ if pc.MatchType != PathMatchTypePrefix {
+ t.Errorf("expected %s got %s", PathMatchTypePrefix, pc.MatchType)
+ }
+
+ if pc.HandlerName != expectedHandlerName {
+ t.Errorf("expected %s got %s", expectedHandlerName, pc.HandlerName)
+ }
+
+ if len(pc.CacheKeyParams) != 1 {
+ t.Errorf("expected %d got %d", 1, len(pc.CacheKeyParams))
+ }
+
+ if len(pc.CacheKeyHeaders) != 1 {
+ t.Errorf("expected %d got %d", 1, len(pc.CacheKeyHeaders))
+ }
+
+ if len(pc.CacheKeyFormFields) != 1 {
+ t.Errorf("expected %d got %d", 1, len(pc.CacheKeyFormFields))
+ }
+
+ if len(pc.RequestHeaders) != 1 {
+ t.Errorf("expected %d got %d", 1, len(pc.RequestHeaders))
+ }
+
+ if len(pc.RequestParams) != 1 {
+ t.Errorf("expected %d got %d", 1, len(pc.RequestParams))
+ }
+
+ if len(pc.ResponseHeaders) != 1 {
+ t.Errorf("expected %d got %d", 1, len(pc.ResponseHeaders))
+ }
+
+ if pc.ResponseCode != 404 {
+ t.Errorf("expected %d got %d", 404, pc.ResponseCode)
+ }
+
+ if pc.ResponseCode != 404 {
+ t.Errorf("expected %d got %d", 404, pc.ResponseCode)
+ }
+
+ if pc.ResponseBody != "trickster" {
+ t.Errorf("expected %s got %s", "trickster", pc.ResponseBody)
+ }
+
+ if !pc.NoMetrics {
+ t.Errorf("expected %t got %t", true, pc.NoMetrics)
+ }
+
+ if pc.OriginConfig == nil {
+ t.Errorf("expected non-nil value you for %s", "OriginConfig")
+ }
+
+ if pc.CollapsedForwardingName != "progressive" || pc.CollapsedForwardingType != CFTypeProgressive {
+ t.Errorf("expected %s got %s", "progressive", pc.CollapsedForwardingName)
+ }
+
+}
diff --git a/internal/config/timeseries_eviction_method.go b/internal/config/timeseries_eviction_method.go
new file mode 100644
index 000000000..a42d8637f
--- /dev/null
+++ b/internal/config/timeseries_eviction_method.go
@@ -0,0 +1,50 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import "strconv"
+
+// TimeseriesEvictionMethod enumerates the methodologies for maintaining time series cache data
+type TimeseriesEvictionMethod int
+
+const (
+ // EvictionMethodOldest indicates that a time series cache object only holds values newer than an explicit date,
+ // called the Oldest Cacheable Timestamp, which is calculated with this formula on each request:
+ // time.Now().Add(-(config.ValueRetentionFactor * query.Step))
+ // This policy is the more performant methodology, because out-of-cache-range determination does not require querying
+ // the cache; thus the cache is only accessed for requests that are pre-determined to be cacheable
+ EvictionMethodOldest = TimeseriesEvictionMethod(iota)
+ // EvictionMethodLRU indicates a that a time series cache object hold up to ValueRetentionFactor number of
+ // unique timestamps, removing the least-recently-used timestamps as necessary to to remain at the ValueRetentionFactor
+ // This policy is the more compute-intensive, since we must maintain an LRU on each timestamp in each cache object,
+ // and retrieve the object from cache on each request
+ EvictionMethodLRU
+)
+
+var timeseriesEvictionMethodNames = map[string]TimeseriesEvictionMethod{
+ "oldest": EvictionMethodOldest,
+ "lru": EvictionMethodLRU,
+}
+
+var timeseriesEvictionMethodValues = map[TimeseriesEvictionMethod]string{
+ EvictionMethodOldest: "oldest",
+ EvictionMethodLRU: "lru",
+}
+
+func (t TimeseriesEvictionMethod) String() string {
+ if v, ok := timeseriesEvictionMethodValues[t]; ok {
+ return v
+ }
+ return strconv.Itoa(int(t))
+}
diff --git a/internal/config/timeseries_eviction_method_test.go b/internal/config/timeseries_eviction_method_test.go
new file mode 100644
index 000000000..d952da19f
--- /dev/null
+++ b/internal/config/timeseries_eviction_method_test.go
@@ -0,0 +1,38 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "testing"
+)
+
+func TestTEMString(t *testing.T) {
+
+ t1 := EvictionMethodLRU
+ t2 := EvictionMethodOldest
+ var t3 TimeseriesEvictionMethod = 3
+
+ if t1.String() != "lru" {
+ t.Errorf("expected %s got %s", "lru", t1.String())
+ }
+
+ if t2.String() != "oldest" {
+ t.Errorf("expected %s got %s", "oldest", t2.String())
+ }
+
+ if t3.String() != "3" {
+ t.Errorf("expected %s got %s", "3", t3.String())
+ }
+
+}
diff --git a/internal/config/tls.go b/internal/config/tls.go
new file mode 100644
index 000000000..921f7824b
--- /dev/null
+++ b/internal/config/tls.go
@@ -0,0 +1,138 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "crypto/tls"
+ "io/ioutil"
+)
+
+// TLSConfig is a collection of TLS-related client and server configurations
+type TLSConfig struct {
+ // FullChainCertPath specifies the path of the file containing the
+ // concatenated server certification and the intermediate certification for the tls endpoint
+ FullChainCertPath string `toml:"full_chain_cert_path"`
+ // PrivateKeyPath specifies the path of the private key file for the tls endpoint
+ PrivateKeyPath string `toml:"private_key_path"`
+ // ServeTLS is set to true once the Cert and Key files have been validated,
+ // indicating the consumer of this config can service requests over TLS
+ ServeTLS bool `toml:"-"`
+ // InsecureSkipVerify indicates that the HTTPS Client in Trickster should bypass
+ // hostname verification for the origin's certificate when proxying requests
+ InsecureSkipVerify bool `toml:"insecure_skip_verify"`
+ // CertificateAuthorities provides a list of custom Certificate Authorities for the upstream origin
+ // which are considered in addition to any system CA's by the Trickster HTTPS Client
+ CertificateAuthorityPaths []string `toml:"certificate_authority_paths"`
+ // ClientCertPath provides the path to the Client Certificate when using Mutual Authorization
+ ClientCertPath string `toml:"client_cert_path"`
+ // ClientKeyPath provides the path to the Client Key when using Mutual Authorization
+ ClientKeyPath string `toml:"client_key_path"`
+}
+
+// DefaultTLSConfig will return a *TLSConfig with the default settings
+func DefaultTLSConfig() *TLSConfig {
+ return &TLSConfig{
+ FullChainCertPath: "",
+ PrivateKeyPath: "",
+ }
+}
+
+// Clone returns an exact copy of the subject *TLSConfig
+func (tc *TLSConfig) Clone() *TLSConfig {
+
+ var caps []string
+ if tc.CertificateAuthorityPaths != nil {
+ caps = make([]string, len(tc.CertificateAuthorityPaths))
+ copy(caps, tc.CertificateAuthorityPaths)
+ }
+
+ return &TLSConfig{
+ FullChainCertPath: tc.FullChainCertPath,
+ PrivateKeyPath: tc.PrivateKeyPath,
+ ServeTLS: tc.ServeTLS,
+ InsecureSkipVerify: tc.InsecureSkipVerify,
+ CertificateAuthorityPaths: caps,
+ ClientCertPath: tc.ClientCertPath,
+ ClientKeyPath: tc.ClientKeyPath,
+ }
+}
+
+func (c *TricksterConfig) verifyTLSConfigs() error {
+
+ for _, oc := range c.Origins {
+
+ if oc.TLS == nil || (oc.TLS.FullChainCertPath == "" || oc.TLS.PrivateKeyPath == "") && (oc.TLS.CertificateAuthorityPaths == nil || len(oc.TLS.CertificateAuthorityPaths) == 0) {
+ continue
+ }
+
+ _, err := ioutil.ReadFile(oc.TLS.FullChainCertPath)
+ if err != nil {
+ return err
+ }
+ _, err = ioutil.ReadFile(oc.TLS.PrivateKeyPath)
+ if err != nil {
+ return err
+ }
+ c.Frontend.ServeTLS = true
+ oc.TLS.ServeTLS = true
+
+ // Verify CA Paths
+ if oc.TLS.CertificateAuthorityPaths != nil && len(oc.TLS.CertificateAuthorityPaths) > 0 {
+ for _, path := range oc.TLS.CertificateAuthorityPaths {
+ _, err = ioutil.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ }
+ }
+
+ }
+ return nil
+}
+
+// TLSCertConfig returns the crypto/tls configuration object with a list of name-bound certs derifed from the running config
+func (c *TricksterConfig) TLSCertConfig() (*tls.Config, error) {
+ var err error
+ if !c.Frontend.ServeTLS {
+ return nil, nil
+ }
+ to := []*OriginConfig{}
+ for _, oc := range c.Origins {
+ if oc.TLS.ServeTLS {
+ to = append(to, oc)
+ }
+ }
+
+ l := len(to)
+ if l == 0 {
+ return nil, nil
+ }
+
+ tlsConfig := &tls.Config{}
+ tlsConfig.Certificates = make([]tls.Certificate, l)
+
+ i := 0
+ for _, tc := range to {
+ tlsConfig.Certificates[i], err = tls.LoadX509KeyPair(tc.TLS.FullChainCertPath, tc.TLS.PrivateKeyPath)
+ if err != nil {
+ return nil, err
+ }
+ i++
+ }
+
+ tlsConfig.BuildNameToCertificate()
+
+ return tlsConfig, nil
+
+}
diff --git a/internal/config/tls_test.go b/internal/config/tls_test.go
new file mode 100644
index 000000000..36b0bbf56
--- /dev/null
+++ b/internal/config/tls_test.go
@@ -0,0 +1,155 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package config
+
+import (
+ "testing"
+)
+
+func TestDefaultTLSConfig(t *testing.T) {
+
+ dc := DefaultTLSConfig()
+ if dc == nil {
+ t.Errorf("expected config named %s", "default")
+ }
+
+ if dc.FullChainCertPath != "" {
+ t.Errorf("expected empty cert path got %s", dc.FullChainCertPath)
+ }
+
+ if dc.PrivateKeyPath != "" {
+ t.Errorf("expected empty key path got %s", dc.PrivateKeyPath)
+ }
+
+}
+
+func tlsConfig(id string) *TLSConfig {
+ return &TLSConfig{
+ FullChainCertPath: "../../testdata/test." + id + ".cert.pem",
+ PrivateKeyPath: "../../testdata/test." + id + ".key.pem",
+ ServeTLS: true,
+ }
+}
+
+func TestVerifyTLSConfigs(t *testing.T) {
+
+ config := NewConfig()
+ tls01 := tlsConfig("01")
+ config.Origins["default"].TLS = tls01
+
+ err := config.verifyTLSConfigs()
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test for error when cert file can't be read
+ tls04 := tlsConfig("04")
+ originalFile := tls04.FullChainCertPath
+ badFile := originalFile + ".nonexistent"
+ tls04.FullChainCertPath = badFile
+ config.Origins["default"].TLS = tls04
+ err = config.verifyTLSConfigs()
+ if err == nil {
+ t.Errorf("expected error for bad file %s", badFile)
+ }
+ tls04.FullChainCertPath = originalFile
+
+ // test for error when key file can't be read
+ originalFile = tls04.PrivateKeyPath
+ badFile = originalFile + ".nonexistent"
+ tls04.PrivateKeyPath = badFile
+ err = config.verifyTLSConfigs()
+ if err == nil {
+ t.Errorf("expected error for bad file %s", badFile)
+ }
+ tls04.PrivateKeyPath = originalFile
+
+ originalFile = "../../testdata/test.rootca.pem"
+ badFile = originalFile + ".nonexistent"
+ // test for more RootCA's to add
+ tls04.CertificateAuthorityPaths = []string{originalFile}
+ err = config.verifyTLSConfigs()
+ if err != nil {
+ t.Error(err)
+ }
+
+ tls04.CertificateAuthorityPaths = []string{badFile}
+ err = config.verifyTLSConfigs()
+ if err == nil {
+ t.Errorf("expected error for bad file %s", badFile)
+ }
+}
+
+func TestProcessTLSConfigs(t *testing.T) {
+
+ a := []string{"-config", "../../testdata/test.full.conf"}
+ err := Load("trickster-test", "0", a)
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestTLSCertConfig(t *testing.T) {
+
+ config := NewConfig()
+
+ // test empty config condition #1 (ServeTLS is false, early bail)
+ n, err := config.TLSCertConfig()
+ if n != nil {
+ t.Errorf("expected nil config, got %d certs", len(n.Certificates))
+ }
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test empty config condition 2 (ServeTLS is true, but there are 0 origins configured)
+ config.Frontend.ServeTLS = true
+ n, err = config.TLSCertConfig()
+ if n != nil {
+ t.Errorf("expected nil config, got %d certs", len(n.Certificates))
+ }
+ if err != nil {
+ t.Error(err)
+ }
+
+ tls01 := tlsConfig("01")
+ config.Frontend.ServeTLS = true
+
+ // test good config
+ config.Origins["default"].TLS = tls01
+ _, err = config.TLSCertConfig()
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test config with key file that has invalid key data
+ expectedErr := "tls: failed to find any PEM data in key input"
+ tls05 := tlsConfig("05")
+ config.Origins["default"].TLS = tls05
+ _, err = config.TLSCertConfig()
+ if err == nil {
+ t.Errorf("expected error: %s", expectedErr)
+ }
+
+ // test config with cert file that has invalid key data
+ expectedErr = "tls: failed to find any PEM data in certificate input"
+ tls06 := tlsConfig("06")
+ config.Origins["default"].TLS = tls06
+ _, err = config.TLSCertConfig()
+ if err == nil {
+ t.Errorf("expected error: %s", expectedErr)
+ }
+
+}
diff --git a/internal/proxy/context/keys.go b/internal/proxy/context/keys.go
new file mode 100644
index 000000000..84fec4715
--- /dev/null
+++ b/internal/proxy/context/keys.go
@@ -0,0 +1,20 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package context
+
+type contextKey int
+
+const (
+ resourcesKey contextKey = iota
+)
diff --git a/internal/proxy/context/resources.go b/internal/proxy/context/resources.go
new file mode 100644
index 000000000..99a166bbe
--- /dev/null
+++ b/internal/proxy/context/resources.go
@@ -0,0 +1,31 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package context
+
+import (
+ "context"
+)
+
+// WithResources returns a copy of the provided context that also includes the OriginConfig, CachingConfig and PathConfig for the request
+func WithResources(ctx context.Context, r interface{}) context.Context {
+ if r != nil {
+ return context.WithValue(ctx, resourcesKey, r)
+ }
+ return ctx
+}
+
+// Resources returns the interface reference to the Request's resources
+func Resources(ctx context.Context) interface{} {
+ return ctx.Value(resourcesKey)
+}
diff --git a/internal/proxy/context/resources_test.go b/internal/proxy/context/resources_test.go
new file mode 100644
index 000000000..12f62deee
--- /dev/null
+++ b/internal/proxy/context/resources_test.go
@@ -0,0 +1,39 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package context
+
+import "testing"
+
+import "context"
+
+type testStruct struct {
+ testField1 bool
+}
+
+func TestResources(t *testing.T) {
+
+ ctx := context.Background()
+
+ // cover nil short circuit case
+ ctx = WithResources(ctx, nil)
+
+ r1 := &testStruct{testField1: true}
+ ctx = WithResources(ctx, r1)
+ r2 := Resources(ctx)
+
+ if !r2.(*testStruct).testField1 {
+ t.Errorf("expected %t got %t", true, r2.(testStruct).testField1)
+ }
+
+}
diff --git a/internal/proxy/engines/access_logs.go b/internal/proxy/engines/access_logs.go
new file mode 100644
index 000000000..d7b409e0d
--- /dev/null
+++ b/internal/proxy/engines/access_logs.go
@@ -0,0 +1,45 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "net/http"
+
+ "github.com/Comcast/trickster/internal/util/log"
+)
+
+func logUpstreamRequest(originName, originType, handlerName, method, path, userAgent string, responseCode, size int, requestDuration float64) {
+ log.Debug("upstream request",
+ log.Pairs{
+ "originName": originName,
+ "originType": originType,
+ "handlerName": handlerName,
+ "method": method,
+ "uri": path,
+ "userAgent": userAgent,
+ "code": responseCode,
+ "size": size,
+ "durationMS": int(requestDuration * 1000),
+ })
+}
+
+func logDownstreamRequest(r *http.Request) {
+ log.Debug("downtream request",
+ log.Pairs{
+ "uri": r.RequestURI,
+ "method": r.Method,
+ "userAgent": r.UserAgent(),
+ "clientIP": r.RemoteAddr,
+ })
+}
diff --git a/internal/proxy/engines/access_logs_test.go b/internal/proxy/engines/access_logs_test.go
new file mode 100644
index 000000000..8df972341
--- /dev/null
+++ b/internal/proxy/engines/access_logs_test.go
@@ -0,0 +1,60 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "net/http"
+ "os"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/util/log"
+)
+
+func TestLogUpstreamRequest(t *testing.T) {
+ fileName := "out.log"
+ // it should create a logger that outputs to a log file ("out.test.log")
+ config.Config = config.NewConfig()
+ config.Main = &config.MainConfig{InstanceID: 0}
+ config.Logging = &config.LoggingConfig{LogFile: fileName, LogLevel: "debug"}
+ log.Init()
+ logUpstreamRequest("testOrigin", "testType", "testHandler", "testMethod", "testPath", "testUserAgent", 200, 0, 1.0)
+ if _, err := os.Stat(fileName); err != nil {
+ t.Errorf(err.Error())
+ }
+ log.Logger.Close()
+ os.Remove(fileName)
+}
+
+func TestLogDownstreamRequest(t *testing.T) {
+ fileName := "out.log"
+ // it should create a logger that outputs to a log file ("out.test.log")
+ config.Config = config.NewConfig()
+ config.Main = &config.MainConfig{InstanceID: 0}
+ config.Logging = &config.LoggingConfig{LogFile: fileName, LogLevel: "debug"}
+ log.Init()
+
+ r, err := http.NewRequest("get", "http://testOrigin", nil)
+ if err != nil {
+ t.Error(err)
+ }
+
+ logDownstreamRequest(r)
+
+ if _, err := os.Stat(fileName); err != nil {
+ t.Errorf(err.Error())
+ }
+ log.Logger.Close()
+ os.Remove(fileName)
+}
diff --git a/internal/proxy/engines/cache.go b/internal/proxy/engines/cache.go
new file mode 100644
index 000000000..c841f1dc3
--- /dev/null
+++ b/internal/proxy/engines/cache.go
@@ -0,0 +1,212 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "mime"
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/golang/snappy"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+ "github.com/Comcast/trickster/internal/util/log"
+)
+
+// QueryCache queries the cache for an HTTPDocument and returns it
+func QueryCache(c cache.Cache, key string, ranges byterange.Ranges) (*HTTPDocument, status.LookupStatus, byterange.Ranges, error) {
+
+ d := &HTTPDocument{}
+ var lookupStatus status.LookupStatus
+ var bytes []byte
+ var err error
+
+ if c.Configuration().CacheType == "memory" {
+ mc := c.(cache.MemoryCache)
+ var ifc interface{}
+ ifc, lookupStatus, err = mc.RetrieveReference(key, true)
+ // normalize any cache miss errors to cache.ErrKNF. We'll get all of them updated so we can remove this code
+ if err != nil && err != cache.ErrKNF && strings.HasSuffix(err.Error(), "not in cache") {
+ err = cache.ErrKNF
+ }
+
+ if err != nil || (lookupStatus != status.LookupStatusHit) {
+ var nr byterange.Ranges
+ if lookupStatus == status.LookupStatusKeyMiss && ranges != nil && len(ranges) > 0 {
+ nr = ranges
+ }
+ return d, lookupStatus, nr, err
+ }
+
+ if ifc != nil {
+ d, _ = ifc.(*HTTPDocument)
+ } else {
+ return d, status.LookupStatusKeyMiss, ranges, err
+ }
+
+ } else {
+
+ bytes, lookupStatus, err = c.Retrieve(key, true)
+ // normalize any cache miss errors to cache.ErrKNF. We'll get all of them updated so we can remove this code
+ if err != nil && err != cache.ErrKNF && strings.HasSuffix(err.Error(), "not in cache") {
+ err = cache.ErrKNF
+ }
+
+ if err != nil || (lookupStatus != status.LookupStatusHit) {
+ var nr byterange.Ranges
+ if lookupStatus == status.LookupStatusKeyMiss && ranges != nil && len(ranges) > 0 {
+ nr = ranges
+
+ }
+ return d, lookupStatus, nr, err
+ }
+
+ var inflate bool
+ // check and remove compression bit
+ if len(bytes) > 0 {
+ if bytes[0] == 1 {
+ inflate = true
+ }
+ bytes = bytes[1:]
+ }
+
+ if inflate {
+ log.Debug("decompressing cached data", log.Pairs{"cacheKey": key})
+ b, err := snappy.Decode(nil, bytes)
+ if err == nil {
+ bytes = b
+ }
+ }
+ _, err = d.UnmarshalMsg(bytes)
+ if err != nil {
+ return d, status.LookupStatusKeyMiss, ranges, err
+ }
+
+ }
+
+ var delta byterange.Ranges
+
+ // Fulfillment is when we have a range stored, but a subsequent user wants the whole body, so
+ // we must inflate the requested range to be the entire object in order to get the correct delta.
+ d.isFulfillment = (d.Ranges != nil && len(d.Ranges) > 0) && (ranges == nil || len(ranges) == 0)
+
+ if d.isFulfillment {
+ ranges = byterange.Ranges{byterange.Range{Start: 0, End: d.ContentLength - 1}}
+ }
+
+ if ranges != nil && len(ranges) > 0 && d.Ranges != nil && len(d.Ranges) > 0 {
+ delta = ranges.CalculateDelta(d.Ranges, d.ContentLength)
+ if delta != nil && len(delta) > 0 {
+ if delta.Equal(ranges) {
+ lookupStatus = status.LookupStatusRangeMiss
+ } else {
+ lookupStatus = status.LookupStatusPartialHit
+ }
+ }
+
+ }
+ return d, lookupStatus, delta, nil
+}
+
+func stripConditionalHeaders(h http.Header) {
+ h.Del(headers.NameIfMatch)
+ h.Del(headers.NameIfUnmodifiedSince)
+ h.Del(headers.NameIfNoneMatch)
+ h.Del(headers.NameIfModifiedSince)
+}
+
+// WriteCache writes an HTTPDocument to the cache
+func WriteCache(c cache.Cache, key string, d *HTTPDocument, ttl time.Duration, compressTypes map[string]bool) error {
+
+ h := http.Header(d.Headers)
+ h.Del(headers.NameDate)
+ h.Del(headers.NameTransferEncoding)
+ h.Del(headers.NameContentRange)
+ h.Del(headers.NameTricksterResult)
+
+ var bytes []byte
+
+ var compress bool
+
+ if ce := http.Header(d.Headers).Get(headers.NameContentEncoding); (ce == "" || ce == "identity") &&
+ (d.CachingPolicy == nil || !d.CachingPolicy.NoTransform) {
+ if mt, _, err := mime.ParseMediaType(d.ContentType); err == nil {
+ if _, ok := compressTypes[mt]; ok {
+ compress = true
+ }
+ }
+ }
+
+ // for memory cache, don't serialize the document, since we can retrieve it by reference.
+ if c.Configuration().CacheType == "memory" {
+ mc := c.(cache.MemoryCache)
+
+ if d != nil {
+ // during unmarshal, these would come back as false, so lets set them as such even for direct access
+ d.rangePartsLoaded = false
+ d.isFulfillment = false
+ d.isLoaded = false
+ d.RangeParts = nil
+
+ if d.CachingPolicy != nil {
+ d.CachingPolicy.ResetClientConditionals()
+ }
+ }
+
+ return mc.StoreReference(key, d, ttl)
+ }
+
+ // for non-memory, we have to seralize the document to a byte slice to store
+ bytes, _ = d.MarshalMsg(nil)
+
+ if compress {
+ log.Debug("compressing cache data", log.Pairs{"cacheKey": key})
+ bytes = append([]byte{1}, snappy.Encode(nil, bytes)...)
+ } else {
+ bytes = append([]byte{0}, bytes...)
+ }
+ return c.Store(key, bytes, ttl)
+
+}
+
+// DocumentFromHTTPResponse returns an HTTPDocument from the provided HTTP Response and Body
+func DocumentFromHTTPResponse(resp *http.Response, body []byte, cp *CachingPolicy) *HTTPDocument {
+ d := &HTTPDocument{}
+ d.StatusCode = resp.StatusCode
+ d.Status = resp.Status
+ d.CachingPolicy = cp
+ d.ContentLength = resp.ContentLength
+
+ if resp.Header != nil {
+ d.Headers = resp.Header.Clone()
+ }
+
+ ct := http.Header(d.Headers).Get(headers.NameContentType)
+ if !strings.HasPrefix(ct, headers.ValueMultipartByteRanges) {
+ d.ContentType = ct
+ }
+
+ if d.StatusCode == http.StatusPartialContent && body != nil && len(body) > 0 {
+ d.ParsePartialContentBody(resp, body)
+ d.FulfillContentBody()
+ } else {
+ d.SetBody(body)
+ }
+
+ return d
+}
diff --git a/internal/proxy/engines/cache_test.go b/internal/proxy/engines/cache_test.go
new file mode 100644
index 000000000..ab8787b89
--- /dev/null
+++ b/internal/proxy/engines/cache_test.go
@@ -0,0 +1,408 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "io/ioutil"
+ "log"
+ "net/http"
+ "net/http/httptest"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ cr "github.com/Comcast/trickster/internal/cache/registration"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+)
+
+const testRangeBody = "This is a test file, to see how the byte range requests work.\n"
+
+func newRangeRequestTestServer() *httptest.Server {
+
+ handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ http.ServeContent(w, r, "", time.Now(),
+ strings.NewReader(testRangeBody))
+ })
+ s := httptest.NewServer(handler)
+ return s
+}
+
+func TestInvalidContentRange(t *testing.T) {
+ _, _, err := byterange.ParseContentRangeHeader("blah")
+ if err == nil {
+ t.Errorf("expected error: %s", `invalid input format`)
+ }
+}
+
+func TestMultiPartByteRange(t *testing.T) {
+
+ // TODO Make this work functionally
+
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+ resp2 := &http.Response{}
+ resp2.Header = make(http.Header)
+ resp2.Header.Add(headers.NameContentLength, "62")
+ resp2.Header.Add(headers.NameContentRange, "bytes 0-10/62")
+ resp2.Header.Add("Content-Type", "multipart/byteranges; boundary=ddffee123")
+ resp2.StatusCode = 200
+ d := DocumentFromHTTPResponse(resp2, []byte("This is a t"), nil)
+
+ ranges := make(byterange.Ranges, 1)
+ ranges[0] = byterange.Range{Start: 5, End: 10}
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error("Expected multi part byte range request to pass, but failed with ", err.Error())
+ }
+}
+
+func TestCacheHitRangeRequest(t *testing.T) {
+ expected := "is a "
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+ resp2 := &http.Response{}
+ resp2.Header = make(http.Header)
+ resp2.Header.Add(headers.NameContentLength, strconv.Itoa(len(testRangeBody)))
+ resp2.StatusCode = 200
+ d := DocumentFromHTTPResponse(resp2, []byte(testRangeBody), nil)
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ ranges := byterange.Ranges{byterange.Range{Start: 5, End: 10}}
+ d2, _, deltas, err := QueryCache(cache, "testKey", ranges)
+ if err != nil {
+ t.Error(err)
+ }
+ if (string(d2.Body[5:10])) != expected {
+ t.Errorf("expected %s got %s", expected, string(d2.Body[5:10]))
+ }
+ if deltas != nil {
+ t.Errorf("updated query range was expected to be empty")
+ }
+}
+
+func TestCacheHitRangeRequest2(t *testing.T) {
+
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ have := byterange.Range{Start: 1, End: 20}
+ cl := int64(len(testRangeBody))
+ rl := (have.End - have.Start) + 1
+ resp2 := &http.Response{}
+ resp2.Header = make(http.Header)
+ resp2.Header.Add(headers.NameContentLength, strconv.FormatInt(rl, 10))
+ resp2.ContentLength = int64(rl)
+ resp2.Header.Add(headers.NameContentRange, have.ContentRangeHeader(cl))
+ resp2.StatusCode = 206
+ d := DocumentFromHTTPResponse(resp2, []byte(testRangeBody[have.Start:have.End+1]), nil)
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ ranges := byterange.Ranges{byterange.Range{Start: 5, End: 10}}
+ d2, _, deltas, err := QueryCache(cache, "testKey", ranges)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if deltas != nil && len(deltas) > 0 {
+ t.Errorf("updated query range was expected to be empty: %v", deltas)
+ }
+ if d2.Ranges[0].Start != 1 || d2.Ranges[0].End != 20 {
+ t.Errorf("expected start %d end %d, got start %d end %d", 1, 20, deltas[0].Start, deltas[0].End)
+ }
+}
+
+func TestCacheHitRangeRequest3(t *testing.T) {
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ have := byterange.Range{Start: 1, End: 20}
+ cl := int64(len(testRangeBody))
+ rl := (have.End - have.Start) + 1
+ resp2 := &http.Response{}
+ resp2.Header = make(http.Header)
+ resp2.Header.Add(headers.NameContentLength, strconv.FormatInt(rl, 10))
+ resp2.ContentLength = int64(rl)
+ resp2.Header.Add(headers.NameContentRange, have.ContentRangeHeader(cl))
+ resp2.StatusCode = 206
+ d := DocumentFromHTTPResponse(resp2, []byte(testRangeBody[have.Start:have.End+1]), nil)
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ qrange := byterange.Ranges{byterange.Range{Start: 5, End: 10}}
+ _, _, deltas, err := QueryCache(cache, "testKey", qrange)
+ if err != nil {
+ t.Error(err)
+ }
+ if deltas != nil && len(deltas) > 0 {
+ t.Error("Expected empty query range got non empty response ", deltas)
+ }
+}
+
+func TestPartialCacheMissRangeRequest(t *testing.T) {
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ have := byterange.Range{Start: 1, End: 9}
+ cl := int64(len(testRangeBody))
+ rl := (have.End - have.Start) + 1
+ resp2 := &http.Response{}
+ resp2.Header = make(http.Header)
+ resp2.Header.Add(headers.NameContentLength, strconv.FormatInt(rl, 10))
+ resp2.ContentLength = int64(rl)
+ resp2.Header.Add(headers.NameContentRange, have.ContentRangeHeader(cl))
+ resp2.StatusCode = 206
+ d := DocumentFromHTTPResponse(resp2, []byte(testRangeBody[have.Start:have.End+1]), nil)
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ ranges := byterange.Ranges{byterange.Range{Start: 5, End: 20}}
+ _, _, deltas, err := QueryCache(cache, "testKey", ranges)
+ if err != nil {
+ t.Error(err)
+ }
+ if deltas == nil || len(deltas) < 1 {
+ t.Errorf("invalid deltas: %v", deltas)
+ } else if deltas[0].Start != 10 ||
+ deltas[0].End != 20 {
+ t.Errorf("expected start %d end %d, got start %d end %d", 10, 20, deltas[0].Start, deltas[0].End)
+ }
+}
+
+func TestFullCacheMissRangeRequest(t *testing.T) {
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+ have := byterange.Range{Start: 1, End: 9}
+ cl := int64(len(testRangeBody))
+ rl := (have.End - have.Start) + 1
+ resp2 := &http.Response{}
+ resp2.Header = make(http.Header)
+ resp2.Header.Add(headers.NameContentLength, strconv.FormatInt(rl, 10))
+ resp2.ContentLength = int64(rl)
+ resp2.Header.Add(headers.NameContentRange, have.ContentRangeHeader(cl))
+ resp2.StatusCode = 206
+ d := DocumentFromHTTPResponse(resp2, []byte(testRangeBody[have.Start:have.End+1]), nil)
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ ranges := byterange.Ranges{byterange.Range{Start: 15, End: 20}}
+ _, _, deltas, err := QueryCache(cache, "testKey", ranges)
+ if err != nil {
+ t.Error(err)
+ }
+ if deltas[0].Start != 15 ||
+ deltas[0].End != 20 {
+ t.Errorf("expected start %d end %d, got start %d end %d", 10, 20, deltas[0].Start, deltas[0].End)
+ }
+}
+
+func TestRangeRequestFromClient(t *testing.T) {
+
+ want := byterange.Ranges{byterange.Range{Start: 15, End: 20}}
+ haves := byterange.Ranges{byterange.Range{Start: 10, End: 25}}
+
+ s := newRangeRequestTestServer()
+ defer s.Close()
+ client := &http.Client{}
+ request, err := http.NewRequest(http.MethodGet, s.URL, nil)
+
+ if err != nil {
+ log.Fatalln(err)
+ }
+ request.Header.Set(headers.NameRange, haves.String())
+ resp, err := client.Do(request)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bytes, _ := ioutil.ReadAll(resp.Body)
+
+ //--------------------------------------
+ err = config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, e2 := cr.GetCache("default")
+ if e2 != nil {
+ t.Error(e2)
+ }
+
+ d := DocumentFromHTTPResponse(resp, bytes, nil)
+ err = WriteCache(cache, "testKey2", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+ _, _, deltas, err := QueryCache(cache, "testKey2", want)
+ if err != nil {
+ t.Error(err)
+ }
+ if deltas != nil && len(deltas) > 0 {
+ t.Errorf("expected cache hit but got cache miss: %s", deltas)
+ }
+ want[0].Start = 20
+ want[0].End = 35
+ _, _, deltas, err = QueryCache(cache, "testKey2", want)
+ if err != nil {
+ t.Error(err)
+ }
+ if deltas == nil {
+ t.Errorf("expected cache miss but got cache hit")
+ }
+ if deltas[0].Start != 26 || deltas[0].End != 35 {
+ t.Errorf("expected start %d end %d, got start %d end %d", 26, 35, deltas[0].Start, deltas[0].End)
+ }
+}
+
+func TestQueryCache(t *testing.T) {
+
+ expected := "1234"
+
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ resp := &http.Response{}
+ resp.Header = make(http.Header)
+ resp.StatusCode = 200
+ resp.Header.Add(headers.NameContentLength, "4")
+ d := DocumentFromHTTPResponse(resp, []byte(expected), nil)
+ d.ContentType = "text/plain"
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ d2, _, _, err := QueryCache(cache, "testKey", nil)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(d2.Body) != string(expected) {
+ t.Errorf("expected %s got %s", string(expected), string(d2.Body))
+ }
+
+ if d2.StatusCode != 200 {
+ t.Errorf("expected %d got %d", 200, d2.StatusCode)
+ }
+
+ _, _, _, err = QueryCache(cache, "testKey2", nil)
+ if err == nil {
+ t.Errorf("expected error")
+ }
+
+ // test marshaling route by making our cache not appear to be a memory cache
+ cache.Remove("testKey")
+ cache.Configuration().CacheType = "test"
+
+ _, _, _, err = QueryCache(cache, "testKey", byterange.Ranges{{Start: 0, End: 1}})
+ if err == nil {
+ t.Errorf("expected error")
+ }
+
+ err = WriteCache(cache, "testKey", d, time.Duration(60)*time.Second, map[string]bool{"text/plain": true})
+ if err != nil {
+ t.Error(err)
+ }
+
+ d2, _, _, err = QueryCache(cache, "testKey", nil)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(d2.Body) != string(expected) {
+ t.Errorf("expected %s got %s", string(expected), string(d2.Body))
+ }
+
+ if d2.StatusCode != 200 {
+ t.Errorf("expected %d got %d", 200, d2.StatusCode)
+ }
+
+}
diff --git a/internal/proxy/engines/caching_policy.go b/internal/proxy/engines/caching_policy.go
new file mode 100644
index 000000000..b9ee6e8ba
--- /dev/null
+++ b/internal/proxy/engines/caching_policy.go
@@ -0,0 +1,419 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "fmt"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+)
+
+//go:generate msgp
+
+// CachingPolicy defines the attributes for determining the cachability of an HTTP object
+type CachingPolicy struct {
+ IsFresh bool `msg:"is_fresh"`
+ NoCache bool `msg:"nocache"`
+ NoTransform bool `msg:"notransform"`
+ FreshnessLifetime int `msg:"freshness_lifetime"`
+ CanRevalidate bool `msg:"can_revalidate"`
+ MustRevalidate bool `msg:"must_revalidate"`
+ LastModified time.Time `msg:"last_modified"`
+ Expires time.Time `msg:"expires"`
+ Date time.Time `msg:"date"`
+ LocalDate time.Time `msg:"local_date"`
+ ETag string `msg:"etag"`
+
+ IsNegativeCache bool `msg:"is_negative_cache"`
+
+ IfNoneMatchValue string `msg:"-"`
+ IfModifiedSinceTime time.Time `msg:"-"`
+ IfUnmodifiedSinceTime time.Time `msg:"-"`
+
+ IsClientConditional bool `msg:"-"`
+ IsClientFresh bool `msg:"-"`
+ HasIfModifiedSince bool `msg:"-"`
+ HasIfUnmodifiedSince bool `msg:"-"`
+ HasIfNoneMatch bool `msg:"-"`
+
+ IfNoneMatchResult bool `msg:"-"`
+}
+
+// Clone returns an exact copy of the Caching Policy
+func (cp *CachingPolicy) Clone() *CachingPolicy {
+ return &CachingPolicy{
+ IsFresh: cp.IsFresh,
+ NoCache: cp.NoCache,
+ NoTransform: cp.NoTransform,
+ FreshnessLifetime: cp.FreshnessLifetime,
+ CanRevalidate: cp.CanRevalidate,
+ MustRevalidate: cp.MustRevalidate,
+ LastModified: cp.LastModified,
+ Expires: cp.Expires,
+ Date: cp.Date,
+ LocalDate: cp.LocalDate,
+ ETag: cp.ETag,
+ IsNegativeCache: cp.IsNegativeCache,
+ IfNoneMatchValue: cp.IfNoneMatchValue,
+ IfModifiedSinceTime: cp.IfModifiedSinceTime,
+ IfUnmodifiedSinceTime: cp.IfUnmodifiedSinceTime,
+ IsClientConditional: cp.IsClientConditional,
+ IsClientFresh: cp.IsClientFresh,
+ HasIfModifiedSince: cp.HasIfModifiedSince,
+ HasIfUnmodifiedSince: cp.HasIfUnmodifiedSince,
+ HasIfNoneMatch: cp.HasIfNoneMatch,
+ IfNoneMatchResult: cp.IfNoneMatchResult,
+ }
+}
+
+// ResetClientConditionals sets the request-specific conditional values of the subject
+// caching policy to false, so as to facilitate reuse of the policy with subsequent requests
+// for the same cache object
+func (cp *CachingPolicy) ResetClientConditionals() {
+ cp.IfNoneMatchValue = ""
+ cp.IfModifiedSinceTime = time.Time{}
+ cp.IfUnmodifiedSinceTime = time.Time{}
+ cp.IsClientConditional = false
+ cp.IsClientFresh = false
+ cp.HasIfModifiedSince = false
+ cp.HasIfUnmodifiedSince = false
+ cp.HasIfNoneMatch = false
+ cp.IfNoneMatchResult = false
+}
+
+// Merge merges the source CachingPolicy into the subject CachingPolicy
+func (cp *CachingPolicy) Merge(src *CachingPolicy) {
+
+ if src == nil {
+ return
+ }
+
+ cp.NoCache = cp.NoCache || src.NoCache
+ cp.NoTransform = cp.NoTransform || src.NoTransform
+
+ cp.IsClientConditional = cp.IsClientConditional || src.IsClientConditional
+ cp.IsClientFresh = cp.IsClientFresh || src.IsClientFresh
+ cp.IsNegativeCache = cp.IsNegativeCache || src.IsNegativeCache
+
+ cp.IsFresh = src.IsFresh
+ cp.FreshnessLifetime = src.FreshnessLifetime
+ cp.CanRevalidate = src.CanRevalidate
+ cp.MustRevalidate = src.MustRevalidate
+ cp.LastModified = src.LastModified
+ cp.Expires = src.Expires
+ cp.Date = src.Date
+ cp.LocalDate = src.LocalDate
+ cp.ETag = src.ETag
+
+ // request policies (e.g., IfModifiedSince) are intentionally omitted,
+ // assuming a response policy is always merged into a request policy
+
+}
+
+// TTL returns a TTL based on the subject caching policy and the provided multiplier and max values
+func (cp *CachingPolicy) TTL(multiplier float64, max time.Duration) time.Duration {
+ var ttl time.Duration = time.Duration(cp.FreshnessLifetime) * time.Second
+ if cp.CanRevalidate {
+ ttl *= time.Duration(multiplier)
+ }
+ if ttl > max {
+ ttl = max
+ }
+ return ttl
+}
+
+func (cp *CachingPolicy) String() string {
+ return fmt.Sprintf(`{ "is_fresh":%t, "no_cache":%t, "no_transform":%t, "freshness_lifetime":%d, "can_revalidate":%t, "must_revalidate":%t,`+
+ ` "last_modified":%d, "expires":%d, "date":%d, "local_date":%d, "etag":"%s", "if_none_match":"%s"`+
+ ` "if_modified_since":%d, "if_unmodified_since":%d, "is_negative_cache":%t }`,
+ cp.IsFresh, cp.NoCache, cp.NoTransform, cp.FreshnessLifetime, cp.CanRevalidate, cp.MustRevalidate, cp.LastModified.Unix(), cp.Expires.Unix(), cp.Date.Unix(), cp.LocalDate.Unix(),
+ cp.ETag, cp.IfNoneMatchValue, cp.IfModifiedSinceTime.Unix(), cp.IfUnmodifiedSinceTime.Unix(), cp.IsNegativeCache)
+}
+
+// GetResponseCachingPolicy examines HTTP response headers for caching headers
+// a returns a CachingPolicy reference
+func GetResponseCachingPolicy(code int, negativeCache map[int]time.Duration, h http.Header) *CachingPolicy {
+
+ cp := &CachingPolicy{LocalDate: time.Now()}
+
+ if d, ok := negativeCache[code]; ok {
+ cp.FreshnessLifetime = int(d.Seconds())
+ cp.Expires = cp.LocalDate.Add(d)
+ cp.IsNegativeCache = true
+ return cp
+ }
+
+ // Do not cache content that includes set-cookie header
+ // Trickster can use PathConfig rules to strip set-cookie if cachablility is needed
+ if v := h.Get(headers.NameSetCookie); v != "" {
+ cp.NoCache = true
+ cp.FreshnessLifetime = -1
+ return cp
+ }
+
+ // Cache-Control has first precedence
+ if v := h.Get(headers.NameCacheControl); v != "" {
+ cp.parseCacheControlDirectives(v)
+ }
+
+ if cp.NoCache {
+ cp.FreshnessLifetime = -1
+ return cp
+ }
+
+ lastModifiedHeader := h.Get(headers.NameLastModified)
+ hasLastModified := lastModifiedHeader != ""
+ expiresHeader := h.Get(headers.NameExpires)
+ hasExpires := expiresHeader != ""
+ eTagHeader := h.Get(headers.NameETag)
+ hasETag := eTagHeader != ""
+
+ if !hasLastModified && !hasExpires && !hasETag && cp.FreshnessLifetime == 0 {
+ cp.NoCache = true
+ cp.FreshnessLifetime = -1
+ return cp
+ }
+
+ // Get the date header or, if it is not found or parsed, set it
+ if v := h.Get(headers.NameDate); v != "" {
+ if date, err := time.Parse(time.RFC1123, v); err != nil {
+ cp.Date = cp.LocalDate
+ h.Set(headers.NameDate, cp.Date.Format(time.RFC1123))
+ } else {
+ cp.Date = date
+ }
+ } else {
+ cp.Date = cp.LocalDate
+ h.Set(headers.NameDate, cp.Date.Format(time.RFC1123))
+ }
+
+ // no Max-Age provided yet, look for expires
+ if cp.FreshnessLifetime == 0 && !cp.MustRevalidate {
+ // if there is an Expires header, respect it
+ if hasExpires {
+ expires, err := time.Parse(time.RFC1123, expiresHeader)
+ if err == nil {
+ cp.Expires = expires
+ if expires.Before(cp.Date) {
+ cp.FreshnessLifetime = -1
+ cp.MustRevalidate = true
+ } else {
+ cp.FreshnessLifetime = int(cp.Expires.Sub(cp.Date).Seconds())
+ }
+ } else {
+ cp.FreshnessLifetime = -1
+ cp.MustRevalidate = true
+ }
+ }
+ }
+
+ if !hasETag && !hasLastModified {
+ cp.CanRevalidate = false
+ return cp
+ }
+
+ cp.CanRevalidate = true
+
+ if hasETag {
+ cp.ETag = eTagHeader
+ }
+
+ if hasLastModified {
+ lm, err := time.Parse(time.RFC1123, lastModifiedHeader)
+ if err != nil {
+ cp.CanRevalidate = false
+ cp.FreshnessLifetime = -1
+ } else {
+ cp.LastModified = lm
+ }
+ }
+
+ // else, if there is a Last-Modified header, set FreshnessLifetime to 20% of age
+ if cp.CanRevalidate && cp.FreshnessLifetime == 0 && !cp.LastModified.IsZero() &&
+ cp.LastModified.Before(cp.Date) && !cp.MustRevalidate {
+ objectAge := int(cp.Date.Sub(cp.LastModified).Seconds())
+ if objectAge > 0 {
+ cp.FreshnessLifetime = objectAge / 5
+ }
+ }
+
+ return cp
+}
+
+var supportedCCD = map[string]bool{
+ headers.ValuePrivate: true,
+ headers.ValueNoCache: true,
+ headers.ValueNoStore: true,
+ headers.ValueMaxAge: false,
+ headers.ValueSharedMaxAge: false,
+ headers.ValueMustRevalidate: false,
+ headers.ValueProxyRevalidate: false,
+}
+
+func (cp *CachingPolicy) parseCacheControlDirectives(directives string) {
+ dl := strings.Split(strings.Replace(strings.ToLower(directives), " ", "", -1), ",")
+ var noCache bool
+ var hasSharedMaxAge bool
+ var foundFreshnessDirective bool
+ for _, d := range dl {
+ var dsub string
+ if i := strings.Index(d, "="); i > 0 {
+ dsub = d[i+1:]
+ d = d[:i]
+ }
+ if v, ok := supportedCCD[d]; ok {
+ noCache = noCache || v
+ }
+ if noCache {
+ cp.NoCache = true
+ cp.FreshnessLifetime = -1
+ return
+ }
+ if d == headers.ValueSharedMaxAge && dsub != "" {
+ foundFreshnessDirective = true
+ secs, err := strconv.Atoi(dsub)
+ if err == nil {
+ hasSharedMaxAge = true
+ cp.FreshnessLifetime = secs
+ }
+ }
+ if (!hasSharedMaxAge) && d == headers.ValueMaxAge && dsub != "" {
+ foundFreshnessDirective = true
+ secs, err := strconv.Atoi(dsub)
+ if err == nil {
+ cp.FreshnessLifetime = secs
+ }
+ }
+ if (d == headers.ValueMustRevalidate || d == headers.ValueProxyRevalidate) || (cp.FreshnessLifetime == 0 && foundFreshnessDirective) {
+ cp.MustRevalidate = true
+ cp.FreshnessLifetime = 0
+ }
+ if d == headers.ValueNoTransform {
+ cp.NoTransform = true
+ }
+ }
+
+}
+
+func hasPragmaNoCache(h http.Header) bool {
+ if v := h.Get(headers.NamePragma); v != "" {
+ return v == headers.ValueNoCache
+ }
+ return false
+}
+
+// GetRequestCachingPolicy examines HTTP request headers for caching headers
+// and true if the corresponding response is OK to cache
+func GetRequestCachingPolicy(h http.Header) *CachingPolicy {
+ cp := &CachingPolicy{LocalDate: time.Now()}
+
+ if hasPragmaNoCache(h) {
+ cp.NoCache = true
+ return cp
+ }
+
+ // Cache-Control has first precedence
+ if v := h.Get(headers.NameCacheControl); v != "" {
+ cp.parseCacheControlDirectives(v)
+ if cp.NoCache {
+ return cp
+ }
+ }
+
+ if v := h.Get(headers.NameIfModifiedSince); v != "" {
+ if date, err := time.Parse(time.RFC1123, v); err == nil {
+ cp.IfModifiedSinceTime = date
+ }
+ }
+
+ if v := h.Get(headers.NameIfUnmodifiedSince); v != "" {
+ if date, err := time.Parse(time.RFC1123, v); err == nil {
+ cp.IfUnmodifiedSinceTime = date
+ }
+ }
+
+ if v := h.Get(headers.NameIfNoneMatch); v != "" {
+ cp.IfNoneMatchValue = v
+ }
+
+ return cp
+}
+
+// ResolveClientConditionals ensures any client conditionals are handled before
+// responding to the client request
+func (cp *CachingPolicy) ResolveClientConditionals(ls status.LookupStatus) {
+
+ cp.IsClientFresh = false
+ if !cp.IsClientConditional {
+ return
+ }
+
+ isClientFresh := true
+ if cp.HasIfNoneMatch {
+ cp.IfNoneMatchResult = CheckIfNoneMatch(cp.ETag, cp.IfNoneMatchValue, ls)
+ isClientFresh = isClientFresh && !cp.IfNoneMatchResult
+ }
+ if cp.HasIfModifiedSince {
+ isClientFresh = isClientFresh && !cp.LastModified.After(cp.IfModifiedSinceTime)
+ }
+ if cp.HasIfUnmodifiedSince {
+ isClientFresh = isClientFresh && cp.LastModified.After(cp.IfUnmodifiedSinceTime)
+ }
+ cp.IsClientFresh = isClientFresh
+}
+
+// ParseClientConditionals inspects the client http request to determine if it includes any conditions
+func (cp *CachingPolicy) ParseClientConditionals() {
+ cp.HasIfNoneMatch = cp.IfNoneMatchValue != ""
+ cp.HasIfModifiedSince = !cp.IfModifiedSinceTime.IsZero()
+ cp.HasIfUnmodifiedSince = !cp.IfUnmodifiedSinceTime.IsZero()
+ cp.IsClientConditional = cp.HasIfNoneMatch || cp.HasIfModifiedSince || cp.HasIfUnmodifiedSince
+}
+
+// CheckIfNoneMatch determines if the provided match value satisfies an "If-None-Match"
+// condition against the cached object. As Trickster is a cache, matching is always weak.
+func CheckIfNoneMatch(etag string, headerValue string, ls status.LookupStatus) bool {
+
+ if etag == "" || headerValue == "" {
+ return etag == headerValue
+ }
+
+ if headerValue == "*" {
+ if ls == status.LookupStatusHit || ls == status.LookupStatusRevalidated {
+ return false
+ }
+ return true
+ }
+
+ parts := strings.Split(headerValue, ",")
+ for _, p := range parts {
+ p = strings.Trim(p, " ")
+ if len(p) > 3 && p[1:2] == "/" {
+ p = p[2:]
+ }
+ if len(p) > 1 && strings.HasPrefix(p, `"`) && strings.HasSuffix(p, `"`) {
+ p = p[1 : len(p)-1]
+ }
+ if p == etag {
+ return false
+ }
+ }
+
+ return true
+}
diff --git a/internal/proxy/engines/caching_policy_gen.go b/internal/proxy/engines/caching_policy_gen.go
new file mode 100644
index 000000000..f64408aad
--- /dev/null
+++ b/internal/proxy/engines/caching_policy_gen.go
@@ -0,0 +1,409 @@
+package engines
+
+// Code generated by github.com/tinylib/msgp DO NOT EDIT.
+
+import (
+ "github.com/tinylib/msgp/msgp"
+)
+
+// DecodeMsg implements msgp.Decodable
+func (z *CachingPolicy) DecodeMsg(dc *msgp.Reader) (err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, err = dc.ReadMapKeyPtr()
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "is_fresh":
+ z.IsFresh, err = dc.ReadBool()
+ if err != nil {
+ return
+ }
+ case "nocache":
+ z.NoCache, err = dc.ReadBool()
+ if err != nil {
+ return
+ }
+ case "notransform":
+ z.NoTransform, err = dc.ReadBool()
+ if err != nil {
+ return
+ }
+ case "freshness_lifetime":
+ z.FreshnessLifetime, err = dc.ReadInt()
+ if err != nil {
+ return
+ }
+ case "can_revalidate":
+ z.CanRevalidate, err = dc.ReadBool()
+ if err != nil {
+ return
+ }
+ case "must_revalidate":
+ z.MustRevalidate, err = dc.ReadBool()
+ if err != nil {
+ return
+ }
+ case "last_modified":
+ z.LastModified, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "expires":
+ z.Expires, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "date":
+ z.Date, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "local_date":
+ z.LocalDate, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "etag":
+ z.ETag, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ case "if_none_match_value":
+ z.IfNoneMatchValue, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ case "if_modified_since_time":
+ z.IfModifiedSinceTime, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "if_unmodified_since_time":
+ z.IfUnmodifiedSinceTime, err = dc.ReadTime()
+ if err != nil {
+ return
+ }
+ case "is_negative_cache":
+ z.IsNegativeCache, err = dc.ReadBool()
+ if err != nil {
+ return
+ }
+ default:
+ err = dc.Skip()
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// EncodeMsg implements msgp.Encodable
+func (z *CachingPolicy) EncodeMsg(en *msgp.Writer) (err error) {
+ // map header, size 15
+ // write "is_fresh"
+ err = en.Append(0x8f, 0xa8, 0x69, 0x73, 0x5f, 0x66, 0x72, 0x65, 0x73, 0x68)
+ if err != nil {
+ return
+ }
+ err = en.WriteBool(z.IsFresh)
+ if err != nil {
+ return
+ }
+ // write "nocache"
+ err = en.Append(0xa7, 0x6e, 0x6f, 0x63, 0x61, 0x63, 0x68, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteBool(z.NoCache)
+ if err != nil {
+ return
+ }
+ // write "notransform"
+ err = en.Append(0xab, 0x6e, 0x6f, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, 0x6d)
+ if err != nil {
+ return
+ }
+ err = en.WriteBool(z.NoTransform)
+ if err != nil {
+ return
+ }
+ // write "freshness_lifetime"
+ err = en.Append(0xb2, 0x66, 0x72, 0x65, 0x73, 0x68, 0x6e, 0x65, 0x73, 0x73, 0x5f, 0x6c, 0x69, 0x66, 0x65, 0x74, 0x69, 0x6d, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteInt(z.FreshnessLifetime)
+ if err != nil {
+ return
+ }
+ // write "can_revalidate"
+ err = en.Append(0xae, 0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x76, 0x61, 0x6c, 0x69, 0x64, 0x61, 0x74, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteBool(z.CanRevalidate)
+ if err != nil {
+ return
+ }
+ // write "must_revalidate"
+ err = en.Append(0xaf, 0x6d, 0x75, 0x73, 0x74, 0x5f, 0x72, 0x65, 0x76, 0x61, 0x6c, 0x69, 0x64, 0x61, 0x74, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteBool(z.MustRevalidate)
+ if err != nil {
+ return
+ }
+ // write "last_modified"
+ err = en.Append(0xad, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.LastModified)
+ if err != nil {
+ return
+ }
+ // write "expires"
+ err = en.Append(0xa7, 0x65, 0x78, 0x70, 0x69, 0x72, 0x65, 0x73)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.Expires)
+ if err != nil {
+ return
+ }
+ // write "date"
+ err = en.Append(0xa4, 0x64, 0x61, 0x74, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.Date)
+ if err != nil {
+ return
+ }
+ // write "local_date"
+ err = en.Append(0xaa, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x5f, 0x64, 0x61, 0x74, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.LocalDate)
+ if err != nil {
+ return
+ }
+ // write "etag"
+ err = en.Append(0xa4, 0x65, 0x74, 0x61, 0x67)
+ if err != nil {
+ return
+ }
+ err = en.WriteString(z.ETag)
+ if err != nil {
+ return
+ }
+ // write "if_none_match_value"
+ err = en.Append(0xb3, 0x69, 0x66, 0x5f, 0x6e, 0x6f, 0x6e, 0x65, 0x5f, 0x6d, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteString(z.IfNoneMatchValue)
+ if err != nil {
+ return
+ }
+ // write "if_modified_since_time"
+ err = en.Append(0xb6, 0x69, 0x66, 0x5f, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64, 0x5f, 0x73, 0x69, 0x6e, 0x63, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.IfModifiedSinceTime)
+ if err != nil {
+ return
+ }
+ // write "if_unmodified_since_time"
+ err = en.Append(0xb8, 0x69, 0x66, 0x5f, 0x75, 0x6e, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64, 0x5f, 0x73, 0x69, 0x6e, 0x63, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteTime(z.IfUnmodifiedSinceTime)
+ if err != nil {
+ return
+ }
+ // write "is_negative_cache"
+ err = en.Append(0xb1, 0x69, 0x73, 0x5f, 0x6e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x5f, 0x63, 0x61, 0x63, 0x68, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteBool(z.IsNegativeCache)
+ if err != nil {
+ return
+ }
+ return
+}
+
+// MarshalMsg implements msgp.Marshaler
+func (z *CachingPolicy) MarshalMsg(b []byte) (o []byte, err error) {
+ o = msgp.Require(b, z.Msgsize())
+ // map header, size 15
+ // string "is_fresh"
+ o = append(o, 0x8f, 0xa8, 0x69, 0x73, 0x5f, 0x66, 0x72, 0x65, 0x73, 0x68)
+ o = msgp.AppendBool(o, z.IsFresh)
+ // string "nocache"
+ o = append(o, 0xa7, 0x6e, 0x6f, 0x63, 0x61, 0x63, 0x68, 0x65)
+ o = msgp.AppendBool(o, z.NoCache)
+ // string "notransform"
+ o = append(o, 0xab, 0x6e, 0x6f, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, 0x6d)
+ o = msgp.AppendBool(o, z.NoTransform)
+ // string "freshness_lifetime"
+ o = append(o, 0xb2, 0x66, 0x72, 0x65, 0x73, 0x68, 0x6e, 0x65, 0x73, 0x73, 0x5f, 0x6c, 0x69, 0x66, 0x65, 0x74, 0x69, 0x6d, 0x65)
+ o = msgp.AppendInt(o, z.FreshnessLifetime)
+ // string "can_revalidate"
+ o = append(o, 0xae, 0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x76, 0x61, 0x6c, 0x69, 0x64, 0x61, 0x74, 0x65)
+ o = msgp.AppendBool(o, z.CanRevalidate)
+ // string "must_revalidate"
+ o = append(o, 0xaf, 0x6d, 0x75, 0x73, 0x74, 0x5f, 0x72, 0x65, 0x76, 0x61, 0x6c, 0x69, 0x64, 0x61, 0x74, 0x65)
+ o = msgp.AppendBool(o, z.MustRevalidate)
+ // string "last_modified"
+ o = append(o, 0xad, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64)
+ o = msgp.AppendTime(o, z.LastModified)
+ // string "expires"
+ o = append(o, 0xa7, 0x65, 0x78, 0x70, 0x69, 0x72, 0x65, 0x73)
+ o = msgp.AppendTime(o, z.Expires)
+ // string "date"
+ o = append(o, 0xa4, 0x64, 0x61, 0x74, 0x65)
+ o = msgp.AppendTime(o, z.Date)
+ // string "local_date"
+ o = append(o, 0xaa, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x5f, 0x64, 0x61, 0x74, 0x65)
+ o = msgp.AppendTime(o, z.LocalDate)
+ // string "etag"
+ o = append(o, 0xa4, 0x65, 0x74, 0x61, 0x67)
+ o = msgp.AppendString(o, z.ETag)
+ // string "if_none_match_value"
+ o = append(o, 0xb3, 0x69, 0x66, 0x5f, 0x6e, 0x6f, 0x6e, 0x65, 0x5f, 0x6d, 0x61, 0x74, 0x63, 0x68, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65)
+ o = msgp.AppendString(o, z.IfNoneMatchValue)
+ // string "if_modified_since_time"
+ o = append(o, 0xb6, 0x69, 0x66, 0x5f, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64, 0x5f, 0x73, 0x69, 0x6e, 0x63, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65)
+ o = msgp.AppendTime(o, z.IfModifiedSinceTime)
+ // string "if_unmodified_since_time"
+ o = append(o, 0xb8, 0x69, 0x66, 0x5f, 0x75, 0x6e, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x69, 0x65, 0x64, 0x5f, 0x73, 0x69, 0x6e, 0x63, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65)
+ o = msgp.AppendTime(o, z.IfUnmodifiedSinceTime)
+ // string "is_negative_cache"
+ o = append(o, 0xb1, 0x69, 0x73, 0x5f, 0x6e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x5f, 0x63, 0x61, 0x63, 0x68, 0x65)
+ o = msgp.AppendBool(o, z.IsNegativeCache)
+ return
+}
+
+// UnmarshalMsg implements msgp.Unmarshaler
+func (z *CachingPolicy) UnmarshalMsg(bts []byte) (o []byte, err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, bts, err = msgp.ReadMapKeyZC(bts)
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "is_fresh":
+ z.IsFresh, bts, err = msgp.ReadBoolBytes(bts)
+ if err != nil {
+ return
+ }
+ case "nocache":
+ z.NoCache, bts, err = msgp.ReadBoolBytes(bts)
+ if err != nil {
+ return
+ }
+ case "notransform":
+ z.NoTransform, bts, err = msgp.ReadBoolBytes(bts)
+ if err != nil {
+ return
+ }
+ case "freshness_lifetime":
+ z.FreshnessLifetime, bts, err = msgp.ReadIntBytes(bts)
+ if err != nil {
+ return
+ }
+ case "can_revalidate":
+ z.CanRevalidate, bts, err = msgp.ReadBoolBytes(bts)
+ if err != nil {
+ return
+ }
+ case "must_revalidate":
+ z.MustRevalidate, bts, err = msgp.ReadBoolBytes(bts)
+ if err != nil {
+ return
+ }
+ case "last_modified":
+ z.LastModified, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "expires":
+ z.Expires, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "date":
+ z.Date, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "local_date":
+ z.LocalDate, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "etag":
+ z.ETag, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ case "if_none_match_value":
+ z.IfNoneMatchValue, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ case "if_modified_since_time":
+ z.IfModifiedSinceTime, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "if_unmodified_since_time":
+ z.IfUnmodifiedSinceTime, bts, err = msgp.ReadTimeBytes(bts)
+ if err != nil {
+ return
+ }
+ case "is_negative_cache":
+ z.IsNegativeCache, bts, err = msgp.ReadBoolBytes(bts)
+ if err != nil {
+ return
+ }
+ default:
+ bts, err = msgp.Skip(bts)
+ if err != nil {
+ return
+ }
+ }
+ }
+ o = bts
+ return
+}
+
+// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
+func (z *CachingPolicy) Msgsize() (s int) {
+ s = 1 + 9 + msgp.BoolSize + 8 + msgp.BoolSize + 12 + msgp.BoolSize + 19 + msgp.IntSize + 15 + msgp.BoolSize + 16 + msgp.BoolSize + 14 + msgp.TimeSize + 8 + msgp.TimeSize + 5 + msgp.TimeSize + 11 + msgp.TimeSize + 5 + msgp.StringPrefixSize + len(z.ETag) + 20 + msgp.StringPrefixSize + len(z.IfNoneMatchValue) + 23 + msgp.TimeSize + 25 + msgp.TimeSize + 18 + msgp.BoolSize
+ return
+}
diff --git a/internal/proxy/engines/caching_policy_gen_test.go b/internal/proxy/engines/caching_policy_gen_test.go
new file mode 100644
index 000000000..fcd37b378
--- /dev/null
+++ b/internal/proxy/engines/caching_policy_gen_test.go
@@ -0,0 +1,123 @@
+package engines
+
+// Code generated by github.com/tinylib/msgp DO NOT EDIT.
+
+import (
+ "bytes"
+ "testing"
+
+ "github.com/tinylib/msgp/msgp"
+)
+
+func TestMarshalUnmarshalCachingPolicy(t *testing.T) {
+ v := CachingPolicy{}
+ bts, err := v.MarshalMsg(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ left, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
+ }
+
+ left, err = msgp.Skip(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
+ }
+}
+
+func BenchmarkMarshalMsgCachingPolicy(b *testing.B) {
+ v := CachingPolicy{}
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.MarshalMsg(nil)
+ }
+}
+
+func BenchmarkAppendMsgCachingPolicy(b *testing.B) {
+ v := CachingPolicy{}
+ bts := make([]byte, 0, v.Msgsize())
+ bts, _ = v.MarshalMsg(bts[0:0])
+ b.SetBytes(int64(len(bts)))
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ bts, _ = v.MarshalMsg(bts[0:0])
+ }
+}
+
+func BenchmarkUnmarshalCachingPolicy(b *testing.B) {
+ v := CachingPolicy{}
+ bts, _ := v.MarshalMsg(nil)
+ b.ReportAllocs()
+ b.SetBytes(int64(len(bts)))
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ _, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
+
+func TestEncodeDecodeCachingPolicy(t *testing.T) {
+ v := CachingPolicy{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+
+ m := v.Msgsize()
+ if buf.Len() > m {
+ t.Logf("WARNING: Msgsize() for %v is inaccurate", v)
+ }
+
+ vn := CachingPolicy{}
+ err := msgp.Decode(&buf, &vn)
+ if err != nil {
+ t.Error(err)
+ }
+
+ buf.Reset()
+ msgp.Encode(&buf, &v)
+ err = msgp.NewReader(&buf).Skip()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkEncodeCachingPolicy(b *testing.B) {
+ v := CachingPolicy{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ en := msgp.NewWriter(msgp.Nowhere)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.EncodeMsg(en)
+ }
+ en.Flush()
+}
+
+func BenchmarkDecodeCachingPolicy(b *testing.B) {
+ v := CachingPolicy{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ rd := msgp.NewEndlessReader(buf.Bytes(), b)
+ dc := msgp.NewReader(rd)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ err := v.DecodeMsg(dc)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
diff --git a/internal/proxy/engines/caching_policy_test.go b/internal/proxy/engines/caching_policy_test.go
new file mode 100644
index 000000000..f1dc4557b
--- /dev/null
+++ b/internal/proxy/engines/caching_policy_test.go
@@ -0,0 +1,313 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "net/http"
+ "strconv"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+)
+
+func TestCachingPolicyClone(t *testing.T) {
+ cp := &CachingPolicy{
+ IsClientFresh: true,
+ }
+ v := cp.Clone().IsClientFresh
+ if !v {
+ t.Errorf("expected %t got %t", true, v)
+ }
+}
+
+func TestMerge(t *testing.T) {
+
+ cp := &CachingPolicy{
+ IsClientFresh: true,
+ }
+
+ cp.Merge(nil)
+ if !cp.IsClientFresh {
+ t.Errorf("expected %t got %t", true, cp.IsClientFresh)
+ }
+
+}
+
+func TestGetResponseCachingPolicy(t *testing.T) {
+
+ now := time.Now().Truncate(time.Second)
+
+ tests := []struct {
+ a http.Header
+ expectedTTL time.Duration
+ }{
+ { // 0 - Cache-Control: no-store
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueNoStore},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 1 - Cache-Control: no-cache
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueNoCache},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 2 - Cache-Control: max-age=300
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueMaxAge + "=300"},
+ },
+ expectedTTL: time.Minute * time.Duration(5),
+ },
+ { // 3 - Cache-Control: max-age= should come back as -1 ttl
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueMaxAge + "="},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 4 - Cache-Control: max-age (no =anything) should come back as 0 ttl
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueMaxAge},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 5 - Cache-Control: private,max-age=300 should be treated as non-cacheable by proxy
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValuePrivate + "," + headers.ValueMaxAge + "=300"},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 6 - Cache-Control: public,max-age=300 should be treated as cacheable by proxy
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValuePublic + "," + headers.ValueMaxAge + "=300"},
+ },
+ expectedTTL: time.Minute * time.Duration(5),
+ },
+ { // 7 - Cache-Control and Expires, Cache-Control should win
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValuePublic + "," + headers.ValueMaxAge + "=300"},
+ headers.NameExpires: []string{"-1"},
+ },
+ expectedTTL: time.Minute * time.Duration(5),
+ },
+ { // 8 - Cache-Control and LastModified, Cache-Control should win
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValuePublic + "," + headers.ValueMaxAge + "=300"},
+ headers.NameLastModified: []string{"Sun, 16 Jun 2019 14:19:04 GMT"},
+ },
+ expectedTTL: time.Minute * time.Duration(5),
+ },
+ { // 9 - Already Expired (could not parse)
+ a: http.Header{
+ headers.NameExpires: []string{"-1"},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 10 - Already Expired (parseable in the past)
+ a: http.Header{
+ headers.NameExpires: []string{"Sun, 16 Jun 2019 14:19:04 GMT"},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 11 - Expires in an hour
+ a: http.Header{
+ headers.NameDate: []string{now.Format(time.RFC1123)},
+ headers.NameExpires: []string{now.Add(time.Hour * time.Duration(1)).Format(time.RFC1123)},
+ },
+ expectedTTL: 1 * time.Hour,
+ },
+ { // 12 - Synthesized TTL from Last Modified
+ a: http.Header{
+ headers.NameDate: []string{now.Format(time.RFC1123)},
+ headers.NameLastModified: []string{now.Add(-time.Hour * time.Duration(5)).Format(time.RFC1123)},
+ },
+ expectedTTL: 1 * time.Hour,
+ },
+ { // 13 - No Cache Control Response Headers
+ a: http.Header{
+ headers.NameDate: []string{now.Format(time.RFC1123)},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 14 - Invalid Date Header Format
+ a: http.Header{
+ headers.NameDate: []string{"1571338193"},
+ headers.NameExpires: []string{"-1"},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 15 - Invalid Date Header Format
+ a: http.Header{
+ headers.NameETag: []string{"etag-test"},
+ },
+ expectedTTL: 0,
+ },
+ { // 16 - Invalid Last Modified Date Header Format
+ a: http.Header{
+ headers.NameLastModified: []string{"1571338193"},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 17 - Must Revalidate
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueMustRevalidate},
+ headers.NameLastModified: []string{"Sun, 16 Jun 2019 14:19:04 GMT"},
+ },
+ expectedTTL: 0,
+ },
+ { // 18 - NoTransform
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueNoTransform},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ { // 19 - Set-Cookie
+ a: http.Header{
+ headers.NameSetCookie: []string{"some-fake-value-for-testing"},
+ },
+ expectedTTL: -1 * time.Second,
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+
+ p := GetResponseCachingPolicy(200, nil, test.a)
+ d := time.Duration(p.FreshnessLifetime) * time.Second
+ if test.expectedTTL != d {
+ t.Errorf("expected ttl of %d got %d", test.expectedTTL, d)
+ }
+ })
+ }
+}
+
+func TestResolveClientConditionalsIUS(t *testing.T) {
+
+ cp := &CachingPolicy{
+ IsClientConditional: true,
+ HasIfUnmodifiedSince: true,
+ LastModified: time.Unix(5, 0),
+ IfUnmodifiedSinceTime: time.Unix(4, 0),
+ }
+ cp.ResolveClientConditionals(status.LookupStatusHit)
+
+ if !cp.IsClientFresh {
+ t.Errorf("expected %t got %t", true, cp.IsClientFresh)
+ }
+
+}
+
+func TestGetResponseCachingPolicyNegativeCache(t *testing.T) {
+ p := GetResponseCachingPolicy(400, map[int]time.Duration{400: 300 * time.Second}, nil)
+ if p.FreshnessLifetime != 300 {
+ t.Errorf("expected ttl of %d got %d", 300, p.FreshnessLifetime)
+ }
+}
+
+func TestGetRequestCacheability(t *testing.T) {
+
+ tests := []struct {
+ a http.Header
+ isCacheable bool
+ }{
+ { // 0 - Cache-Control: no-store
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueNoStore},
+ },
+ isCacheable: false,
+ },
+ { // 1 - Cache-Control: no-cache
+ a: http.Header{
+ headers.NameCacheControl: []string{headers.ValueNoCache},
+ },
+ isCacheable: false,
+ },
+ { // 2 - No Cache Control Request Headers
+ a: http.Header{},
+ isCacheable: true,
+ },
+ { // 3 - Pragma: NoCache
+ a: http.Header{
+ headers.NamePragma: []string{headers.ValueNoCache},
+ },
+ isCacheable: false,
+ },
+ { // 4 - IMS
+ a: http.Header{
+ headers.NameIfModifiedSince: []string{"Sun, 16 Jun 2019 14:19:04 GMT"},
+ },
+ isCacheable: true,
+ },
+ { // 5 - IUS
+ a: http.Header{
+ headers.NameIfUnmodifiedSince: []string{"Sun, 16 Jun 2019 14:19:04 GMT"},
+ },
+ isCacheable: true,
+ },
+ { // 6 - INM
+ a: http.Header{
+ headers.NameIfNoneMatch: []string{"test-string"},
+ },
+ isCacheable: true,
+ },
+ { // 7 - IM
+ a: http.Header{
+ headers.NameIfMatch: []string{"test-string"},
+ },
+ isCacheable: true,
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ p := GetRequestCachingPolicy(test.a)
+ ic := !p.NoCache
+ if ic != test.isCacheable {
+ t.Errorf("mismatch isCacheable expected %v got %v", test.isCacheable, ic)
+ }
+ })
+ }
+
+}
+
+func TestCheckIfNoneMatch(t *testing.T) {
+
+ res := CheckIfNoneMatch("", "", status.LookupStatusHit)
+ if !res {
+ t.Errorf("expected %t got %t", true, res)
+ }
+
+ res = CheckIfNoneMatch("test", "*", status.LookupStatusHit)
+ if res {
+ t.Errorf("expected %t got %t", false, res)
+ }
+
+ res = CheckIfNoneMatch("test", "*", status.LookupStatusKeyMiss)
+ if !res {
+ t.Errorf("expected %t got %t", true, res)
+ }
+
+ res = CheckIfNoneMatch("test", "test", status.LookupStatusHit)
+ if res {
+ t.Errorf("expected %t got %t", false, res)
+ }
+
+ res = CheckIfNoneMatch("test", "w/test", status.LookupStatusHit)
+ if res {
+ t.Errorf("expected %t got %t", false, res)
+ }
+
+}
diff --git a/internal/proxy/engines/client_test.go b/internal/proxy/engines/client_test.go
new file mode 100644
index 000000000..7abdc9f88
--- /dev/null
+++ b/internal/proxy/engines/client_test.go
@@ -0,0 +1,833 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "net/http"
+ "net/url"
+ "sort"
+ "strconv"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/errors"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ tt "github.com/Comcast/trickster/internal/proxy/timeconv"
+ "github.com/Comcast/trickster/internal/proxy/urls"
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/pkg/sort/times"
+
+ "github.com/prometheus/common/model"
+)
+
+// Prometheus API
+const (
+ APIPath = "/api/v1/"
+ mnQueryRange = "query_range"
+ mnQuery = "query"
+ mnLabels = "labels"
+ mnLabel = "label"
+ mnSeries = "series"
+ mnTargets = "targets"
+ mnRules = "rules"
+ mnAlerts = "alerts"
+ mnAlertManagers = "alertmanagers"
+ mnStatus = "status"
+)
+
+// Common URL Parameter Names
+const (
+ upQuery = "query"
+ upStart = "start"
+ upEnd = "end"
+ upStep = "step"
+ upTime = "time"
+ upMatch = "match[]"
+)
+
+// Client Implements Proxy Client Interface
+type TestClient struct {
+ name string
+ config *config.OriginConfig
+ cache cache.Cache
+ webClient *http.Client
+
+ fftime time.Time
+ InstantCacheKey string
+ RangeCacheKey string
+
+ handlers map[string]http.Handler
+ handlersRegistered bool
+}
+
+func (c *TestClient) registerHandlers() {
+ c.handlersRegistered = true
+ c.handlers = make(map[string]http.Handler)
+ // This is the registry of handlers that Trickster supports for Prometheus,
+ // and are able to be referenced by name (map key) in Config Files
+ c.handlers["health"] = http.HandlerFunc(c.HealthHandler)
+ c.handlers[mnQueryRange] = http.HandlerFunc(c.QueryRangeHandler)
+ c.handlers[mnQuery] = http.HandlerFunc(c.QueryHandler)
+ c.handlers[mnSeries] = http.HandlerFunc(c.SeriesHandler)
+ c.handlers["proxycache"] = http.HandlerFunc(c.QueryHandler)
+ c.handlers["proxy"] = http.HandlerFunc(c.ProxyHandler)
+}
+
+// Handlers returns a map of the HTTP Handlers the client has registered
+func (c *TestClient) Handlers() map[string]http.Handler {
+ if !c.handlersRegistered {
+ c.registerHandlers()
+ }
+ return c.handlers
+}
+
+// DefaultPathConfigs returns the default PathConfigs for the given OriginType
+func (c *TestClient) DefaultPathConfigs(oc *config.OriginConfig) map[string]*config.PathConfig {
+
+ paths := map[string]*config.PathConfig{
+
+ APIPath + mnQueryRange: {
+ Path: APIPath + mnQueryRange,
+ HandlerName: mnQueryRange,
+ Methods: []string{http.MethodGet, http.MethodPost},
+ CacheKeyParams: []string{upQuery, upStep},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ ResponseHeaders: map[string]string{headers.NameCacheControl: fmt.Sprintf("%s=%d", headers.ValueSharedMaxAge, 86400)},
+ },
+
+ APIPath + mnQuery: {
+ Path: APIPath + mnQuery,
+ HandlerName: mnQuery,
+ Methods: []string{http.MethodGet, http.MethodPost},
+ CacheKeyParams: []string{upQuery, upTime},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ ResponseHeaders: map[string]string{headers.NameCacheControl: fmt.Sprintf("%s=%d", headers.ValueSharedMaxAge, 30)},
+ },
+
+ APIPath + mnSeries: {
+ Path: APIPath + mnSeries,
+ HandlerName: mnSeries,
+ Methods: []string{http.MethodGet, http.MethodPost},
+ CacheKeyParams: []string{upMatch, upStart, upEnd},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnLabels: {
+ Path: APIPath + mnLabels,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet, http.MethodPost},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnLabel: {
+ Path: APIPath + mnLabel,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnTargets: {
+ Path: APIPath + mnTargets,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnRules: {
+ Path: APIPath + mnRules,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnAlerts: {
+ Path: APIPath + mnAlerts,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnAlertManagers: {
+ Path: APIPath + mnAlertManagers,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath + mnStatus: {
+ Path: APIPath + mnStatus,
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ CacheKeyParams: []string{},
+ CacheKeyHeaders: []string{headers.NameAuthorization},
+ },
+
+ APIPath: {
+ Path: APIPath,
+ HandlerName: "proxy",
+ Methods: []string{http.MethodGet, http.MethodPost},
+ },
+
+ "/opc": {
+ Path: "/opc",
+ HandlerName: "proxycache",
+ Methods: []string{http.MethodGet},
+ },
+
+ "/": {
+ Path: "/",
+ HandlerName: "proxy",
+ Methods: []string{http.MethodGet, http.MethodPost},
+ },
+ }
+
+ oc.Paths = paths
+ oc.FastForwardPath = paths[APIPath+mnQuery]
+
+ return paths
+
+}
+
+// Configuration returns the upstream Configuration for this Client
+func (c *TestClient) Configuration() *config.OriginConfig {
+ return c.config
+}
+
+// SetCache sets the cache object the client will use for caching origin data
+func (c *TestClient) SetCache(cc cache.Cache) {
+ c.cache = cc
+}
+
+// HTTPClient returns the HTTP Client for this origin
+func (c *TestClient) HTTPClient() *http.Client {
+ return c.webClient
+}
+
+// Name returns the name of the upstream Configuration proxied by the Client
+func (c *TestClient) Name() string {
+ return c.name
+}
+
+// Cache returns and handle to the Cache instance used by the Client
+func (c *TestClient) Cache() cache.Cache {
+ return c.cache
+}
+
+// parseTime converts a query time URL parameter to time.Time.
+// Copied from https://github.com/prometheus/prometheus/blob/master/web/api/v1/api.go
+func parseTime(s string) (time.Time, error) {
+ if t, err := strconv.ParseFloat(s, 64); err == nil {
+ s, ns := math.Modf(t)
+ ns = math.Round(ns*1000) / 1000
+ return time.Unix(int64(s), int64(ns*float64(time.Second))), nil
+ }
+ if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
+ return t, nil
+ }
+ return time.Time{}, fmt.Errorf("cannot parse %q to a valid timestamp", s)
+}
+
+// parseDuration parses prometheus step parameters, which can be float64 or durations like 1d, 5m, etc
+// the proxy.ParseDuration handles the second kind, and the float64's are handled here
+func parseDuration(input string) (time.Duration, error) {
+ v, err := strconv.ParseFloat(input, 64)
+ if err != nil {
+ return tt.ParseDuration(input)
+ }
+ // assume v is in seconds
+ return time.Duration(int64(v)) * time.Second, nil
+}
+
+// ParseTimeRangeQuery parses the key parts of a TimeRangeQuery from the inbound HTTP Request
+func (c *TestClient) ParseTimeRangeQuery(r *http.Request) (*timeseries.TimeRangeQuery, error) {
+
+ trq := ×eries.TimeRangeQuery{Extent: timeseries.Extent{}}
+ qp := r.URL.Query()
+
+ trq.Statement = qp.Get(upQuery)
+ if trq.Statement == "" {
+ return nil, errors.MissingURLParam(upQuery)
+ }
+
+ if p := qp.Get(upStart); p != "" {
+ t, err := parseTime(p)
+ if err != nil {
+ return nil, err
+ }
+ trq.Extent.Start = t
+ } else {
+ return nil, errors.MissingURLParam(upStart)
+ }
+
+ if p := qp.Get(upEnd); p != "" {
+ t, err := parseTime(p)
+ if err != nil {
+ return nil, err
+ }
+ trq.Extent.End = t
+ } else {
+ return nil, errors.MissingURLParam(upEnd)
+ }
+
+ if p := qp.Get(upStep); p != "" {
+ step, err := parseDuration(p)
+ if err != nil {
+ return nil, err
+ }
+ trq.Step = step
+ } else {
+ return nil, errors.MissingURLParam(upStep)
+ }
+
+ if strings.Contains(trq.Statement, " offset ") {
+ trq.IsOffset = true
+ trq.FastForwardDisable = true
+ }
+
+ return trq, nil
+}
+
+// BaseURL returns a URL in the form of scheme://host/path based on the proxy configuration
+func (c *TestClient) BaseURL() *url.URL {
+ u := &url.URL{}
+ u.Scheme = c.config.Scheme
+ u.Host = c.config.Host
+ u.Path = c.config.PathPrefix
+ return u
+}
+
+// BuildUpstreamURL will merge the downstream request with the BaseURL to construct the full upstream URL
+func (c *TestClient) BuildUpstreamURL(r *http.Request) *url.URL {
+ u := c.BaseURL()
+
+ if strings.HasPrefix(r.URL.Path, "/"+c.name+"/") {
+ u.Path += strings.Replace(r.URL.Path, "/"+c.name+"/", "/", 1)
+ } else {
+ u.Path += r.URL.Path
+ }
+
+ u.RawQuery = r.URL.RawQuery
+ u.Fragment = r.URL.Fragment
+ u.User = r.URL.User
+ return u
+}
+
+// SetExtent will change the upstream request query to use the provided Extent
+func (c *TestClient) SetExtent(r *http.Request, trq *timeseries.TimeRangeQuery, extent *timeseries.Extent) {
+ params := r.URL.Query()
+ params.Set(upStart, strconv.FormatInt(extent.Start.Unix(), 10))
+ params.Set(upEnd, strconv.FormatInt(extent.End.Unix(), 10))
+ r.URL.RawQuery = params.Encode()
+}
+
+// FastForwardURL returns the url to fetch the Fast Forward value based on a timerange url
+func (c *TestClient) FastForwardURL(r *http.Request) (*url.URL, error) {
+
+ u := urls.Clone(r.URL)
+
+ if strings.HasSuffix(u.Path, "/query_range") {
+ u.Path = u.Path[0 : len(u.Path)-6]
+ }
+
+ // let the test client have a way to throw an error
+ if strings.Contains(u.RawQuery, "throw_ffurl_error=1") {
+ return nil, fmt.Errorf("This is an intentional test error: %s", ":)")
+ }
+
+ p := u.Query()
+ p.Del(upStart)
+ p.Del(upEnd)
+ p.Del(upStep)
+
+ if c.fftime.IsZero() {
+ c.fftime = time.Now()
+ }
+ p.Set("time", strconv.FormatInt(c.fftime.Unix(), 10))
+
+ u.RawQuery = p.Encode()
+
+ return u, nil
+}
+
+// VectorEnvelope represents a Vector response object from the Prometheus HTTP API
+type VectorEnvelope struct {
+ Status string `json:"status"`
+ Data VectorData `json:"data"`
+}
+
+// VectorData represents the Data body of a Vector response object from the Prometheus HTTP API
+type VectorData struct {
+ ResultType string `json:"resultType"`
+ Result model.Vector `json:"result"`
+}
+
+// MatrixEnvelope represents a Matrix response object from the Prometheus HTTP API
+type MatrixEnvelope struct {
+ Status string `json:"status"`
+ Data MatrixData `json:"data"`
+ ExtentList timeseries.ExtentList `json:"extents,omitempty"`
+ StepDuration time.Duration `json:"step,omitempty"`
+
+ timestamps map[time.Time]bool // tracks unique timestamps in the matrix data
+ tslist times.Times
+ isSorted bool // tracks if the matrix data is currently sorted
+ isCounted bool // tracks if timestamps slice is up-to-date
+}
+
+// MatrixData represents the Data body of a Matrix response object from the Prometheus HTTP API
+type MatrixData struct {
+ ResultType string `json:"resultType"`
+ Result model.Matrix `json:"result"`
+}
+
+// MarshalTimeseries converts a Timeseries into a JSON blob
+func (c *TestClient) MarshalTimeseries(ts timeseries.Timeseries) ([]byte, error) {
+ // Marshal the Envelope back to a json object for Cache Storage
+ if c.RangeCacheKey == "failkey" {
+ return nil, fmt.Errorf("generic failure for testing purposes (key: %s)", c.RangeCacheKey)
+ }
+ return json.Marshal(ts)
+}
+
+// UnmarshalTimeseries converts a JSON blob into a Timeseries
+func (c *TestClient) UnmarshalTimeseries(data []byte) (timeseries.Timeseries, error) {
+ me := &MatrixEnvelope{}
+ err := json.Unmarshal(data, &me)
+ return me, err
+}
+
+// UnmarshalInstantaneous converts a JSON blob into an Instantaneous Data Point
+func (c *TestClient) UnmarshalInstantaneous(data []byte) (timeseries.Timeseries, error) {
+ ve := &VectorEnvelope{}
+ err := json.Unmarshal(data, &ve)
+ if err != nil {
+ return nil, err
+ }
+ return ve.ToMatrix(), nil
+}
+
+// ToMatrix converts a VectorEnvelope to a MatrixEnvelope
+func (ve *VectorEnvelope) ToMatrix() *MatrixEnvelope {
+ me := &MatrixEnvelope{}
+ me.Status = ve.Status
+ me.Data = MatrixData{
+ ResultType: "matrix",
+ Result: make(model.Matrix, 0, len(ve.Data.Result)),
+ }
+ var ts time.Time
+ for _, v := range ve.Data.Result {
+ v.Timestamp = model.TimeFromUnix(v.Timestamp.Unix()) // Round to nearest Second
+ ts = v.Timestamp.Time()
+ me.Data.Result = append(me.Data.Result, &model.SampleStream{Metric: v.Metric, Values: []model.SamplePair{{Timestamp: v.Timestamp, Value: v.Value}}})
+ }
+ me.ExtentList = timeseries.ExtentList{timeseries.Extent{Start: ts, End: ts}}
+ return me
+}
+
+// Step returns the step for the Timeseries
+func (me *MatrixEnvelope) Step() time.Duration {
+ return me.StepDuration
+}
+
+// SetStep sets the step for the Timeseries
+func (me *MatrixEnvelope) SetStep(step time.Duration) {
+ me.StepDuration = step
+}
+
+// Merge merges the provided Timeseries list into the base Timeseries (in the order provided) and optionally sorts the merged Timeseries
+func (me *MatrixEnvelope) Merge(sort bool, collection ...timeseries.Timeseries) {
+ meMetrics := make(map[string]*model.SampleStream)
+ for _, s := range me.Data.Result {
+ meMetrics[s.Metric.String()] = s
+ }
+ for _, ts := range collection {
+ if ts != nil {
+ me2 := ts.(*MatrixEnvelope)
+ for _, s := range me2.Data.Result {
+ name := s.Metric.String()
+ if _, ok := meMetrics[name]; !ok {
+ meMetrics[name] = s
+ me.Data.Result = append(me.Data.Result, s)
+ continue
+ }
+ meMetrics[name].Values = append(meMetrics[name].Values, s.Values...)
+ }
+ me.ExtentList = append(me.ExtentList, me2.ExtentList...)
+ }
+ }
+ me.ExtentList = me.ExtentList.Compress(me.StepDuration)
+ me.isSorted = false
+ me.isCounted = false
+ if sort {
+ me.Sort()
+ }
+}
+
+// Clone returns a perfect copy of the base Timeseries
+func (me *MatrixEnvelope) Clone() timeseries.Timeseries {
+ resMe := &MatrixEnvelope{
+ isCounted: me.isCounted,
+ isSorted: me.isSorted,
+ tslist: make(times.Times, len(me.tslist)),
+ timestamps: make(map[time.Time]bool),
+ Status: me.Status,
+ Data: MatrixData{
+ ResultType: me.Data.ResultType,
+ Result: make(model.Matrix, 0, len(me.Data.Result)),
+ },
+ StepDuration: me.StepDuration,
+ ExtentList: make(timeseries.ExtentList, len(me.ExtentList)),
+ }
+ copy(resMe.ExtentList, me.ExtentList)
+ copy(resMe.tslist, me.tslist)
+
+ for k, v := range me.timestamps {
+ resMe.timestamps[k] = v
+ }
+
+ for _, ss := range me.Data.Result {
+ newSS := &model.SampleStream{Metric: ss.Metric}
+ newSS.Values = ss.Values[:]
+ resMe.Data.Result = append(resMe.Data.Result, newSS)
+ }
+ return resMe
+}
+
+// CropToSize reduces the number of elements in the Timeseries to the provided count, by evicting elements
+// using a least-recently-used methodology. Any timestamps newer than the provided time are removed before
+// sizing, in order to support backfill tolerance. The provided extent will be marked as used during crop.
+func (me *MatrixEnvelope) CropToSize(sz int, t time.Time, lur timeseries.Extent) {
+ me.isCounted = false
+ me.isSorted = false
+ x := len(me.ExtentList)
+ // The Series has no extents, so no need to do anything
+ if x < 1 {
+ me.Data.Result = model.Matrix{}
+ me.ExtentList = timeseries.ExtentList{}
+ return
+ }
+
+ // Crop to the Backfill Tolerance Value if needed
+ if me.ExtentList[x-1].End.After(t) {
+ me.CropToRange(timeseries.Extent{Start: me.ExtentList[0].Start, End: t})
+ }
+
+ tc := me.TimestampCount()
+ if len(me.Data.Result) == 0 || tc <= sz {
+ return
+ }
+
+ el := timeseries.ExtentListLRU(me.ExtentList).UpdateLastUsed(lur, me.StepDuration)
+ sort.Sort(el)
+
+ rc := tc - sz // # of required timestamps we must delete to meet the rentention policy
+ removals := make(map[time.Time]bool)
+ done := false
+ var ok bool
+
+ for _, x := range el {
+ for ts := x.Start; !x.End.Before(ts) && !done; ts = ts.Add(me.StepDuration) {
+ if _, ok = me.timestamps[ts]; ok {
+ removals[ts] = true
+ done = len(removals) >= rc
+ }
+ }
+ if done {
+ break
+ }
+ }
+
+ for _, s := range me.Data.Result {
+ tmp := s.Values[:0]
+ for _, r := range s.Values {
+ t = r.Timestamp.Time()
+ if _, ok := removals[t]; !ok {
+ tmp = append(tmp, r)
+ }
+ }
+ s.Values = tmp
+ }
+
+ tl := times.FromMap(removals)
+ sort.Sort(tl)
+ for _, t := range tl {
+ for i, e := range el {
+ if e.StartsAt(t) {
+ el[i].Start = e.Start.Add(me.StepDuration)
+ }
+ }
+ }
+
+ me.ExtentList = timeseries.ExtentList(el).Compress(me.StepDuration)
+ me.Sort()
+}
+
+// CropToRange reduces the Timeseries down to timestamps contained within the provided Extents (inclusive).
+// CropToRange assumes the base Timeseries is already sorted, and will corrupt an unsorted Timeseries
+func (me *MatrixEnvelope) CropToRange(e timeseries.Extent) {
+ me.isCounted = false
+ x := len(me.ExtentList)
+ // The Series has no extents, so no need to do anything
+ if x < 1 {
+ me.Data.Result = model.Matrix{}
+ me.ExtentList = timeseries.ExtentList{}
+ return
+ }
+
+ // if the extent of the series is entirely outside the extent of the crop range, return empty set and bail
+ if me.ExtentList.OutsideOf(e) {
+ me.Data.Result = model.Matrix{}
+ me.ExtentList = timeseries.ExtentList{}
+ return
+ }
+
+ // if the series extent is entirely inside the extent of the crop range, simply adjust down its ExtentList
+ if me.ExtentList.InsideOf(e) {
+ if me.ValueCount() == 0 {
+ me.Data.Result = model.Matrix{}
+ }
+ me.ExtentList = me.ExtentList.Crop(e)
+ return
+ }
+
+ if len(me.Data.Result) == 0 {
+ me.ExtentList = me.ExtentList.Crop(e)
+ return
+ }
+
+ deletes := make(map[int]bool)
+
+ for i, s := range me.Data.Result {
+ start := -1
+ end := -1
+ for j, val := range s.Values {
+ t := val.Timestamp.Time()
+ if t.Equal(e.End) {
+ // for cases where the first element is the only qualifying element,
+ // start must be incremented or an empty response is returned
+ if j == 0 || t.Equal(e.Start) || start == -1 {
+ start = j
+ }
+ end = j + 1
+ break
+ }
+ if t.After(e.End) {
+ end = j
+ break
+ }
+ if t.Before(e.Start) {
+ continue
+ }
+ if start == -1 && (t.Equal(e.Start) || (e.End.After(t) && t.After(e.Start))) {
+ start = j
+ }
+ }
+ if start != -1 && len(s.Values) > 0 {
+ if end == -1 {
+ end = len(s.Values)
+ }
+ me.Data.Result[i].Values = s.Values[start:end]
+ } else {
+ deletes[i] = true
+ }
+ }
+ if len(deletes) > 0 {
+ tmp := me.Data.Result[:0]
+ for i, r := range me.Data.Result {
+ if _, ok := deletes[i]; !ok {
+ tmp = append(tmp, r)
+ }
+ }
+ me.Data.Result = tmp
+ }
+ me.ExtentList = me.ExtentList.Crop(e)
+}
+
+// Sort sorts all Values in each Series chronologically by their timestamp
+func (me *MatrixEnvelope) Sort() {
+
+ if me.isSorted {
+ return
+ }
+
+ tsm := map[time.Time]bool{}
+
+ for i, s := range me.Data.Result { // []SampleStream
+ m := make(map[time.Time]model.SamplePair)
+ for _, v := range s.Values { // []SamplePair
+ t := v.Timestamp.Time()
+ tsm[t] = true
+ m[t] = v
+ }
+ keys := make(times.Times, 0, len(m))
+ for key := range m {
+ keys = append(keys, key)
+ }
+ sort.Sort(keys)
+ sm := make([]model.SamplePair, 0, len(keys))
+ for _, key := range keys {
+ sm = append(sm, m[key])
+ }
+ me.Data.Result[i].Values = sm
+ }
+
+ sort.Sort(me.ExtentList)
+
+ me.timestamps = tsm
+ me.tslist = times.FromMap(tsm)
+ me.isCounted = true
+ me.isSorted = true
+}
+
+func (me *MatrixEnvelope) updateTimestamps() {
+ if me.isCounted {
+ return
+ }
+ m := make(map[time.Time]bool)
+ for _, s := range me.Data.Result { // []SampleStream
+ for _, v := range s.Values { // []SamplePair
+ t := v.Timestamp.Time()
+ m[t] = true
+ }
+ }
+ me.timestamps = m
+ me.tslist = times.FromMap(m)
+ me.isCounted = true
+}
+
+// SetExtents overwrites a Timeseries's known extents with the provided extent list
+func (me *MatrixEnvelope) SetExtents(extents timeseries.ExtentList) {
+ me.isCounted = false
+ me.ExtentList = extents
+}
+
+// Extents returns the Timeseries's ExentList
+func (me *MatrixEnvelope) Extents() timeseries.ExtentList {
+ return me.ExtentList
+}
+
+// TimestampCount returns the number of unique timestamps across the timeseries
+func (me *MatrixEnvelope) TimestampCount() int {
+ me.updateTimestamps()
+ return len(me.timestamps)
+}
+
+// SeriesCount returns the number of individual Series in the Timeseries object
+func (me *MatrixEnvelope) SeriesCount() int {
+ return len(me.Data.Result)
+}
+
+// ValueCount returns the count of all values across all Series in the Timeseries object
+func (me *MatrixEnvelope) ValueCount() int {
+ c := 0
+ for i := range me.Data.Result {
+ c += len(me.Data.Result[i].Values)
+ }
+ return c
+}
+
+func (c *TestClient) HealthHandler(w http.ResponseWriter, r *http.Request) {
+ u := c.BaseURL()
+ u.Path += APIPath + mnLabels
+ DoProxy(w, r)
+}
+
+func (c *TestClient) QueryRangeHandler(w http.ResponseWriter, r *http.Request) {
+
+ //rsc := request.NewResources(c.config, c.path
+
+ r.URL = c.BuildUpstreamURL(r)
+ DeltaProxyCacheRequest(w, r)
+}
+
+func (c *TestClient) QueryHandler(w http.ResponseWriter, r *http.Request) {
+ r.URL = c.BuildUpstreamURL(r)
+ ObjectProxyCacheRequest(w, r)
+}
+
+func (c *TestClient) SeriesHandler(w http.ResponseWriter, r *http.Request) {
+ r.URL = c.BuildUpstreamURL(r)
+ ObjectProxyCacheRequest(w, r)
+}
+
+func (c *TestClient) ProxyHandler(w http.ResponseWriter, r *http.Request) {
+ DoProxy(w, r)
+}
+
+func testResultHeaderPartMatch(header http.Header, kvp map[string]string) error {
+ if len(kvp) == 0 {
+ return nil
+ }
+ if header == nil || len(header) == 0 {
+ return fmt.Errorf("missing response headers%s", "")
+ }
+
+ if h, ok := header["X-Trickster-Result"]; ok {
+ res := strings.Join(h, "; ")
+ for k, v := range kvp {
+ if !strings.Contains(res, fmt.Sprintf("; %s=%s", k, v)) && strings.Index(res, fmt.Sprintf("%s=%s", k, v)) != 0 {
+ return fmt.Errorf("invalid status, expected %s=%s in %s", k, v, h)
+ }
+ }
+ } else {
+ return fmt.Errorf("missing X-Trickster-Result header%s", "")
+ }
+
+ return nil
+}
+
+func testStatusCodeMatch(have, expected int) error {
+ if have != expected {
+ return fmt.Errorf("expected http status %d got %d", expected, have)
+ }
+ return nil
+}
+
+func testStringMatch(have, expected string) error {
+ if have != expected {
+ return fmt.Errorf("expected string `%s` got `%s`", expected, have)
+ }
+ return nil
+}
+
+// Size returns the approximate memory utilization in bytes of the timeseries
+func (me *MatrixEnvelope) Size() int {
+
+ c := 0
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+ for i := range me.Data.Result {
+ wg.Add(1)
+ go func(s *model.SampleStream) {
+ mtx.Lock()
+ c += (len(s.Values) * 16) + len(s.Metric.String())
+ mtx.Unlock()
+ wg.Done()
+ }(me.Data.Result[i])
+ }
+ wg.Wait()
+ return c
+}
diff --git a/internal/proxy/engines/deltaproxycache.go b/internal/proxy/engines/deltaproxycache.go
new file mode 100644
index 000000000..9c4c4ceef
--- /dev/null
+++ b/internal/proxy/engines/deltaproxycache.go
@@ -0,0 +1,382 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "net/http"
+ "net/url"
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ tctx "github.com/Comcast/trickster/internal/proxy/context"
+ "github.com/Comcast/trickster/internal/proxy/origins"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/internal/util/metrics"
+ "github.com/Comcast/trickster/pkg/locks"
+)
+
+// DeltaProxyCache is used for Time Series Acceleration, and not used for normal HTTP Object Caching
+
+// DeltaProxyCacheRequest identifies the gaps between the cache and a new timeseries request,
+// requests the gaps from the origin server and returns the reconstituted dataset to the downstream request
+// while caching the results for subsequent requests of the same data
+func DeltaProxyCacheRequest(w http.ResponseWriter, r *http.Request) {
+
+ rsc := request.GetResources(r)
+
+ oc := rsc.OriginConfig
+ pc := rsc.PathConfig
+ cache := rsc.CacheClient
+ cc := rsc.CacheConfig
+
+ client := rsc.OriginClient.(origins.TimeseriesClient)
+
+ trq, err := client.ParseTimeRangeQuery(r)
+ if err != nil {
+ // err may simply mean incompatible query (e.g., non-select), so just proxy
+ DoProxy(w, r)
+ return
+ }
+
+ var cacheStatus status.LookupStatus
+
+ pr := newProxyRequest(r, w)
+ trq.FastForwardDisable = oc.FastForwardDisable || trq.FastForwardDisable
+ trq.NormalizeExtent()
+
+ // this is used to ensure the head of the cache respects the BackFill Tolerance
+ bf := timeseries.Extent{Start: time.Unix(0, 0), End: trq.Extent.End}
+
+ if !trq.IsOffset && oc.BackfillTolerance > 0 {
+ bf.End = bf.End.Add(-oc.BackfillTolerance)
+ }
+
+ now := time.Now()
+
+ OldestRetainedTimestamp := time.Time{}
+ if oc.TimeseriesEvictionMethod == config.EvictionMethodOldest {
+ OldestRetainedTimestamp = now.Truncate(trq.Step).Add(-(trq.Step * oc.TimeseriesRetention))
+ if trq.Extent.End.Before(OldestRetainedTimestamp) {
+ log.Debug("timerange end is too early to consider caching", log.Pairs{"oldestRetainedTimestamp": OldestRetainedTimestamp, "step": trq.Step, "retention": oc.TimeseriesRetention})
+ DoProxy(w, r)
+ return
+ }
+ if trq.Extent.Start.After(bf.End) {
+ log.Debug("timerange is too new to cache due to backfill tolerance", log.Pairs{"backFillToleranceSecs": oc.BackfillToleranceSecs, "newestRetainedTimestamp": bf.End, "queryStart": trq.Extent.Start})
+ DoProxy(w, r)
+ return
+ }
+ }
+
+ client.SetExtent(r, trq, &trq.Extent)
+ key := oc.CacheKeyPrefix + "." + pr.DeriveCacheKey(trq.TemplateURL, "")
+
+ locks.Acquire(key)
+
+ // this is used to determine if Fast Forward should be activated for this request
+ normalizedNow := ×eries.TimeRangeQuery{
+ Extent: timeseries.Extent{Start: time.Unix(0, 0), End: now},
+ Step: trq.Step,
+ }
+ normalizedNow.NormalizeExtent()
+
+ var cts timeseries.Timeseries
+ var doc *HTTPDocument
+ var elapsed time.Duration
+
+ coReq := GetRequestCachingPolicy(r.Header)
+ if coReq.NoCache {
+ cacheStatus = status.LookupStatusPurge
+ cache.Remove(key)
+ cts, doc, elapsed, err = fetchTimeseries(pr, trq, client)
+ if err != nil {
+ recordDPCResult(r, status.LookupStatusProxyError, doc.StatusCode, r.URL.Path, "", elapsed.Seconds(), nil, doc.Headers)
+ Respond(w, doc.StatusCode, doc.Headers, doc.Body)
+ locks.Release(key)
+ return // fetchTimeseries logs the error
+ }
+ } else {
+ doc, cacheStatus, _, err = QueryCache(cache, key, nil)
+ if cacheStatus == status.LookupStatusKeyMiss {
+ cts, doc, elapsed, err = fetchTimeseries(pr, trq, client)
+ if err != nil {
+ recordDPCResult(r, status.LookupStatusProxyError, doc.StatusCode, r.URL.Path, "", elapsed.Seconds(), nil, doc.Headers)
+ Respond(w, doc.StatusCode, doc.Headers, doc.Body)
+ locks.Release(key)
+ return // fetchTimeseries logs the error
+ }
+ } else {
+ // Load the Cached Timeseries
+ if doc == nil {
+ err = errors.New("empty document body")
+ } else {
+ if cc.CacheType == "memory" {
+ cts = doc.timeseries
+ } else {
+ cts, err = client.UnmarshalTimeseries(doc.Body)
+ }
+ }
+ if err != nil {
+ log.Error("cache object unmarshaling failed", log.Pairs{"key": key, "originName": client.Name()})
+ cache.Remove(key)
+ cts, doc, elapsed, err = fetchTimeseries(pr, trq, client)
+ if err != nil {
+ recordDPCResult(r, status.LookupStatusProxyError, doc.StatusCode, r.URL.Path, "", elapsed.Seconds(), nil, doc.Headers)
+ Respond(w, doc.StatusCode, doc.Headers, doc.Body)
+ locks.Release(key)
+ return // fetchTimeseries logs the error
+ }
+ } else {
+ if oc.TimeseriesEvictionMethod == config.EvictionMethodLRU {
+ el := cts.Extents()
+ tsc := cts.TimestampCount()
+ if tsc > 0 &&
+ tsc >= oc.TimeseriesRetentionFactor {
+ if trq.Extent.End.Before(el[0].Start) {
+ log.Debug("timerange end is too early to consider caching", log.Pairs{"step": trq.Step, "retention": oc.TimeseriesRetention})
+ locks.Release(key)
+ DoProxy(w, r)
+ return
+ }
+ if trq.Extent.Start.After(el[len(el)-1].End) {
+ log.Debug("timerange is too new to cache due to backfill tolerance", log.Pairs{"backFillToleranceSecs": oc.BackfillToleranceSecs, "newestRetainedTimestamp": bf.End, "queryStart": trq.Extent.Start})
+ locks.Release(key)
+ DoProxy(w, r)
+ return
+ }
+ }
+ }
+ cacheStatus = status.LookupStatusPartialHit
+ }
+ }
+ }
+
+ // Find the ranges that we want, but which are not currently cached
+ var missRanges timeseries.ExtentList
+ if cacheStatus == status.LookupStatusPartialHit {
+ missRanges = trq.CalculateDeltas(cts.Extents())
+ }
+
+ if len(missRanges) == 0 && cacheStatus == status.LookupStatusPartialHit {
+ // on full cache hit, elapsed records the time taken to query the cache and definitively conclude that it is a full cache hit
+ elapsed = time.Since(now)
+ cacheStatus = status.LookupStatusHit
+ } else if len(missRanges) == 1 && missRanges[0].Start.Equal(trq.Extent.Start) && missRanges[0].End.Equal(trq.Extent.End) {
+ cacheStatus = status.LookupStatusRangeMiss
+ }
+
+ ffStatus := "off"
+
+ var ffURL *url.URL
+ // if the step resolution <= Fast Forward TTL, then no need to even try Fast Forward
+ if !trq.FastForwardDisable {
+ if trq.Step > oc.FastForwardTTL {
+ ffURL, err = client.FastForwardURL(r)
+ if err != nil || ffURL == nil {
+ ffStatus = "err"
+ trq.FastForwardDisable = true
+ }
+ } else {
+ trq.FastForwardDisable = true
+ }
+ }
+
+ dpStatus := log.Pairs{"cacheKey": key, "cacheStatus": cacheStatus, "reqStart": trq.Extent.Start.Unix(), "reqEnd": trq.Extent.End.Unix()}
+ if len(missRanges) > 0 {
+ dpStatus["extentsFetched"] = timeseries.ExtentList(missRanges).String()
+ }
+
+ // maintain a list of timeseries to merge into the main timeseries
+ mts := make([]timeseries.Timeseries, 0, len(missRanges))
+ wg := sync.WaitGroup{}
+ appendLock := sync.Mutex{}
+ uncachedValueCount := 0
+
+ // iterate each time range that the client needs and fetch from the upstream origin
+ for i := range missRanges {
+ wg.Add(1)
+ // This fetches the gaps from the origin and adds their datasets to the merge list
+ go func(e *timeseries.Extent, rq *proxyRequest) {
+ rq.Request = rq.WithContext(tctx.WithResources(r.Context(), request.NewResources(oc, pc, cc, cache, client)))
+ client.SetExtent(rq.Request, trq, e)
+ body, resp, _ := rq.Fetch()
+ if resp.StatusCode == http.StatusOK && len(body) > 0 {
+ nts, err := client.UnmarshalTimeseries(body)
+ if err != nil {
+ log.Error("proxy object unmarshaling failed", log.Pairs{"body": string(body)})
+ wg.Done()
+ return
+ }
+ uncachedValueCount += nts.ValueCount()
+ nts.SetStep(trq.Step)
+ nts.SetExtents([]timeseries.Extent{*e})
+ appendLock.Lock()
+ mts = append(mts, nts)
+ appendLock.Unlock()
+ }
+ wg.Done()
+ }(&missRanges[i], pr.Clone())
+ }
+
+ var hasFastForwardData bool
+ var ffts timeseries.Timeseries
+ // Only fast forward if configured and the user request is for the absolute latest datapoint
+ if (!trq.FastForwardDisable) && (trq.Extent.End.Equal(normalizedNow.Extent.End)) && ffURL.Scheme != "" {
+ wg.Add(1)
+ rs := request.NewResources(oc, oc.FastForwardPath, cc, cache, client)
+ rs.AlternateCacheTTL = oc.FastForwardTTL
+ req := r.Clone(tctx.WithResources(context.Background(), rs))
+ go func() {
+ // create a new context that uses the fast forward path config instead of the time series path config
+ req.URL = ffURL
+ body, resp, isHit := FetchViaObjectProxyCache(req)
+ if resp.StatusCode == http.StatusOK && len(body) > 0 {
+ ffts, err = client.UnmarshalInstantaneous(body)
+ if err != nil {
+ ffStatus = "err"
+ log.Error("proxy object unmarshaling failed", log.Pairs{"body": string(body)})
+ wg.Done()
+ return
+ }
+ ffts.SetStep(trq.Step)
+ x := ffts.Extents()
+ if isHit {
+ ffStatus = "hit"
+ } else {
+ ffStatus = "miss"
+ }
+ hasFastForwardData = len(x) > 0 && x[0].End.After(trq.Extent.End)
+ } else {
+ ffStatus = "err"
+ }
+ wg.Done()
+ }()
+ }
+
+ wg.Wait()
+
+ // Merge the new delta timeseries into the cached timeseries
+ if len(mts) > 0 {
+ // on a partial hit, elapsed should record the amount of time waiting for all upstream requests to complete
+ elapsed = time.Since(now)
+ cts.Merge(true, mts...)
+ }
+
+ // cts is the cacheable time series, rts is the user's response timeseries
+ rts := cts.Clone()
+
+ // if it was a cache key miss, there is no need to undergo Crop since the extents are identical
+ if cacheStatus != status.LookupStatusKeyMiss {
+ rts.CropToRange(trq.Extent)
+ }
+ cachedValueCount := rts.ValueCount() - uncachedValueCount
+
+ if uncachedValueCount > 0 {
+ metrics.ProxyRequestElements.WithLabelValues(oc.Name, oc.OriginType, "uncached", r.URL.Path).Add(float64(uncachedValueCount))
+ }
+
+ if cachedValueCount > 0 {
+ metrics.ProxyRequestElements.WithLabelValues(oc.Name, oc.OriginType, "cached", r.URL.Path).Add(float64(cachedValueCount))
+ }
+
+ // Merge Fast Forward data if present. This must be done after the Downstream Crop since
+ // the cropped extent was normalized to stepboundaries and would remove fast forward data
+ // If the fast forward data point is older (e.g. cached) than the last datapoint in the returned time series, it will not be merged
+ if hasFastForwardData && len(ffts.Extents()) == 1 && ffts.Extents()[0].Start.Truncate(time.Second).After(normalizedNow.Extent.End) {
+ rts.Merge(false, ffts)
+ }
+ rts.SetExtents(nil) // so they are not included in the client response json
+ rts.SetStep(0)
+ rdata, err := client.MarshalTimeseries(rts)
+ rh := http.Header(doc.Headers).Clone()
+
+ switch cacheStatus {
+ case status.LookupStatusKeyMiss, status.LookupStatusPartialHit, status.LookupStatusRangeMiss:
+ wg.Add(1)
+ // Write the newly-merged object back to the cache
+ go func() {
+ defer wg.Done()
+ // Crop the Cache Object down to the Sample Size or Age Retention Policy and the Backfill Tolerance before storing to cache
+ switch oc.TimeseriesEvictionMethod {
+ case config.EvictionMethodLRU:
+ cts.CropToSize(oc.TimeseriesRetentionFactor, bf.End, trq.Extent)
+ default:
+ cts.CropToRange(timeseries.Extent{End: bf.End, Start: OldestRetainedTimestamp})
+ }
+ // Don't cache datasets with empty extents (everything was cropped so there is nothing to cache)
+ if len(cts.Extents()) > 0 {
+ if cc.CacheType == "memory" {
+ doc.timeseries = cts
+ } else {
+ cdata, err := client.MarshalTimeseries(cts)
+ if err != nil {
+ locks.Release(key)
+ return
+ }
+ doc.Body = cdata
+ }
+ WriteCache(cache, key, doc, oc.TimeseriesTTL, oc.CompressableTypes)
+ }
+ }()
+ }
+
+ // Respond to the user. Using the response headers from a Delta Response, so as to not map conflict with cacheData on WriteCache
+ logDeltaRoutine(dpStatus)
+ recordDPCResult(r, cacheStatus, doc.StatusCode, r.URL.Path, ffStatus, elapsed.Seconds(), missRanges, rh)
+ Respond(w, doc.StatusCode, rh, rdata)
+
+ wg.Wait()
+ locks.Release(key)
+}
+
+func logDeltaRoutine(p log.Pairs) { log.Debug("delta routine completed", p) }
+
+func fetchTimeseries(pr *proxyRequest, trq *timeseries.TimeRangeQuery, client origins.TimeseriesClient) (timeseries.Timeseries, *HTTPDocument, time.Duration, error) {
+
+ body, resp, elapsed := pr.Fetch()
+
+ d := &HTTPDocument{
+ Status: resp.Status,
+ StatusCode: resp.StatusCode,
+ Headers: resp.Header,
+ Body: body,
+ }
+
+ if resp.StatusCode != 200 {
+ log.Error("unexpected upstream response", log.Pairs{"statusCode": resp.StatusCode})
+ return nil, d, time.Duration(0), fmt.Errorf("Unexpected Upstream Response")
+ }
+
+ ts, err := client.UnmarshalTimeseries(body)
+ if err != nil {
+ log.Error("proxy object unmarshaling failed", log.Pairs{"body": string(body)})
+ return nil, d, time.Duration(0), err
+ }
+
+ ts.SetExtents([]timeseries.Extent{trq.Extent})
+ ts.SetStep(trq.Step)
+
+ return ts, d, elapsed, nil
+}
+
+func recordDPCResult(r *http.Request, cacheStatus status.LookupStatus, httpStatus int, path, ffStatus string, elapsed float64, needed []timeseries.Extent, header http.Header) {
+ recordResults(r, "DeltaProxyCache", cacheStatus, httpStatus, path, ffStatus, elapsed, timeseries.ExtentList(needed), header)
+}
diff --git a/internal/proxy/engines/deltaproxycache_test.go b/internal/proxy/engines/deltaproxycache_test.go
new file mode 100644
index 000000000..fe6ec6d35
--- /dev/null
+++ b/internal/proxy/engines/deltaproxycache_test.go
@@ -0,0 +1,1577 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ cr "github.com/Comcast/trickster/internal/cache/registration"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/timeseries"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+ "github.com/Comcast/trickster/pkg/promsim"
+)
+
+// test queries
+const (
+ queryReturnsOKNoLatency = "some_query_here{latency_ms=0,range_latency_ms=0}"
+ queryReturnsBadPayload = "some_query_here{invalid_response_body=1,latency_ms=0,range_latency_ms=0}"
+ queryReturnsBadRequest = "some_query_here{status_code=400,latency_ms=0,range_latency_ms=0}"
+ queryReturnsBadGateway = "some_query_here{status_code=502,latency_ms=0,range_latency_ms=0}"
+)
+
+var testConfigFile string
+
+func setupTestHarnessDPC() (*httptest.Server, *httptest.ResponseRecorder, *http.Request, *request.Resources, error) {
+
+ client := &TestClient{}
+ ts, w, r, hc, err := tu.NewTestInstance(testConfigFile, client.DefaultPathConfigs, 200, "", nil, "promsim", "/api/v1/query_range", "debug")
+ if err != nil {
+ return nil, nil, nil, nil, fmt.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ rsc := request.GetResources(r)
+ rsc.OriginClient = client
+ pc := rsc.PathConfig
+
+ if pc == nil {
+ return nil, nil, nil, nil, fmt.Errorf("could not find path %s", "/api/v1/query_range")
+ }
+
+ oc := rsc.OriginConfig
+ cc := rsc.CacheClient
+ oc.HTTPClient = hc
+
+ client.cache = cc
+ client.webClient = hc
+ client.config = oc
+
+ pc.CacheKeyParams = []string{"rangeKey"}
+ pc.CacheKeyParams = []string{"instantKey"}
+
+ return ts, w, r, rsc, nil
+}
+
+func TestDeltaProxyCacheRequestMissThenHit(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+
+ oc.FastForwardDisable = true
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // get cache hit coverage too by repeating:
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "hit"})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestDeltaProxyCacheRequestAllItemsTooNew(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+ oc.BackfillToleranceSecs = 600
+ oc.BackfillTolerance = time.Second * time.Duration(oc.BackfillToleranceSecs)
+
+ step := time.Duration(300) * time.Second
+ end := time.Now()
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(5) * time.Minute), End: end}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extr.Start, extr.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if resp.Header.Get("status") != "" {
+ t.Errorf("status header should not be present. Found with value %s", resp.Header.Get("stattus"))
+ }
+
+ // ensure the request was sent through the proxy instead of the DeltaProxyCache
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestRemoveStale(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // get cache hit coverage too by repeating:
+
+ oc.TimeseriesRetention = 10
+
+ extr = timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: now}
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ _, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestRemoveStaleLRU(t *testing.T) {
+
+ testConfigFile = "../../../testdata/test.cache-lru.conf"
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ testConfigFile = ""
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // get cache hit coverage too by repeating:
+
+ oc.TimeseriesRetention = 10
+
+ extr = timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: now}
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ w = httptest.NewRecorder()
+
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ _, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestMarshalFailure(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+
+ rsc.CacheConfig.CacheType = "test"
+ client.RangeCacheKey = "failkey"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), "")
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func normalizeTime(t time.Time, d time.Duration) time.Time {
+ return time.Unix((t.Unix()/int64(d.Seconds()))*int64(d.Seconds()), 0)
+ //return t.Truncate(d)
+}
+
+func TestDeltaProxyCacheRequestPartialHit(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ client.RangeCacheKey = "test-range-key-phit"
+ client.InstantCacheKey = "test-instant-key-phit"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: normalizeTime(extr.Start, step), End: normalizeTime(extr.End, step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s&rk=%s&ik=%s", int(step.Seconds()),
+ extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency, client.RangeCacheKey, client.InstantCacheKey)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test partial hit (needing an upper fragment)
+ phitStart := normalizeTime(extr.End.Add(step), step)
+ extr.End = extr.End.Add(time.Duration(1) * time.Hour) // Extend the top by 1 hour to generate partial hit
+ extn.End = normalizeTime(extr.End, step)
+
+ expectedFetched := fmt.Sprintf("[%d:%d]", phitStart.Unix(), extn.End.Unix())
+ expected, _, _ = promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s&rk=%s&ik=%s", int(step.Seconds()),
+ extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency, client.RangeCacheKey, client.InstantCacheKey)
+
+ r.URL = u
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "phit"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test partial hit (needing a lower fragment)
+ phitEnd := extn.Start.Add(-step)
+ extr.Start = extr.Start.Add(time.Duration(-1) * time.Hour)
+ extn.Start = normalizeTime(extr.Start, step)
+
+ expectedFetched = fmt.Sprintf("[%d:%d]", extn.Start.Unix(), phitEnd.Unix())
+ expected, _, _ = promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s&rk=%s&ik=%s", int(step.Seconds()),
+ extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency, client.RangeCacheKey, client.InstantCacheKey)
+
+ r.URL = u
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "phit"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test partial hit (needing both upper and lower fragments)
+ phitEnd = normalizeTime(extr.Start.Add(-step), step)
+ phitStart = normalizeTime(extr.End.Add(step), step)
+
+ extr.Start = extr.Start.Add(time.Duration(-1) * time.Hour)
+ extn.Start = normalizeTime(extr.Start, step)
+ extr.End = extr.End.Add(time.Duration(1) * time.Hour) // Extend the top by 1 hour to generate partial hit
+ extn.End = normalizeTime(extr.End, step)
+
+ expectedFetched = fmt.Sprintf("[%d:%d,%d:%d]",
+ extn.Start.Unix(), phitEnd.Unix(), phitStart.Unix(), extn.End.Unix())
+
+ expected, _, _ = promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s&rk=%s&ik=%s", int(step.Seconds()),
+ extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency, client.RangeCacheKey, client.InstantCacheKey)
+
+ r.URL = u
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "phit"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestDeltayProxyCacheRequestDeltaFetchError(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ client.RangeCacheKey = "testkey"
+ client.InstantCacheKey = "testInstantKey"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: normalizeTime(extr.Start, step), End: normalizeTime(extr.End, step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // test partial hit (needing an upper fragment)
+ //phitStart := extr.End.Add(step)
+ extr.End = extr.End.Add(time.Duration(1) * time.Hour) // Extend the top by 1 hour to generate partial hit
+ extn.End = extr.End.Truncate(step)
+
+ //expectedFetched := fmt.Sprintf("[%d:%d]", phitStart.Truncate(step).Unix(), extn.End.Unix())
+ promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ client.InstantCacheKey = "foo1"
+ client.RangeCacheKey = "foo2"
+
+ // Switch to the failed query.
+ u.RawQuery = fmt.Sprintf("instantKey=foo1&step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadGateway)
+
+ r.URL = u
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadGateway)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "proxy-error"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ // if err != nil {
+ // t.Error(err)
+ // }
+
+}
+
+func TestDeltaProxyCacheRequestRangeMiss(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(3600) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // Test Range Miss Low End
+
+ extr.Start = extr.Start.Add(time.Duration(-3) * time.Hour)
+ extn.Start = extr.Start.Truncate(step)
+ extr.End = extr.Start.Add(time.Duration(1) * time.Hour)
+ extn.End = extr.End.Truncate(step)
+
+ expectedFetched := fmt.Sprintf("[%d:%d]", extn.Start.Unix(), extn.End.Unix())
+ expected, _, _ = promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ r.URL = u
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "rmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // Test Range Miss High End
+
+ extr.Start = now.Add(time.Duration(-10) * time.Hour)
+ extn.Start = extr.Start.Truncate(step)
+ extr.End = now.Add(time.Duration(-8) * time.Hour)
+ extn.End = extr.End.Truncate(step)
+
+ expectedFetched = fmt.Sprintf("[%d:%d", extn.Start.Unix(), extn.End.Unix())
+ expected, _, _ = promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+ r.URL = u
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "rmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestDeltaProxyCacheRequestFastForward(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+ rsc.CacheConfig.CacheType = "test"
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+
+ client.InstantCacheKey = "test-dpc-ff-key-instant"
+ client.RangeCacheKey = "test-dpc-ff-key-range"
+
+ oc.FastForwardDisable = false
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ client.fftime = now.Truncate(oc.FastForwardTTL)
+
+ extr := timeseries.Extent{Start: now.Add(-time.Duration(12) * time.Hour), End: now}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("instantKey=%s&rangeKey=%s&step=%d&start=%d&end=%d&query=%s",
+ client.InstantCacheKey, client.RangeCacheKey, int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ expectedMatrix, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+ em, err := client.UnmarshalTimeseries([]byte(expectedMatrix))
+ if err != nil {
+ t.Error(err)
+ }
+ em.SetExtents(timeseries.ExtentList{extn})
+
+ expectedVector, _, _ := promsim.GetInstantData(queryReturnsOKNoLatency, client.fftime)
+ ev, err := client.UnmarshalInstantaneous([]byte(expectedVector))
+ if err != nil {
+ t.Error(err)
+ }
+ ev.SetStep(step)
+
+ if len(ev.Extents()) == 1 && len(em.Extents()) > 0 && ev.Extents()[0].Start.Truncate(time.Second).After(em.Extents()[0].End) {
+ em.Merge(false, ev)
+ }
+
+ em.SetExtents(nil)
+ b, err := client.MarshalTimeseries(em)
+ if err != nil {
+ t.Error(err)
+ }
+
+ expected := string(b)
+
+ cr.LoadCachesFromConfig()
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"ffstatus": "miss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // do it again and look for a cache hit on the timeseries and fast forward
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "hit"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"ffstatus": "hit"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // instantKey := oc.Host + "." + md5.Checksum(strings.Replace(u.Path, "_range", "", -1)+client.InstantCacheKey) + ".sz"
+ // client.cache.Remove(instantKey)
+
+ // u.RawQuery = fmt.Sprintf("instantKey=%s&rangeKey=%s&step=%d&start=%d&end=%d&query=%s",
+ // client.InstantCacheKey, client.RangeCacheKey, int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadPayload)
+
+ // w = httptest.NewRecorder()
+ // client.QueryRangeHandler(w, r)
+ // resp = w.Result()
+
+ // err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ // if err != nil {
+ // t.Error(err)
+ // }
+
+ // err = testResultHeaderPartMatch(resp.Header, map[string]string{"ffstatus": "err"})
+ // if err != nil {
+ // t.Error(err)
+ // }
+
+ // // Now test a Response Code error
+
+ // u.RawQuery = fmt.Sprintf("instantKey=%s&rangeKey=%s&step=%d&start=%d&end=%d&query=%s",
+ // client.InstantCacheKey+"1", client.RangeCacheKey+"1", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadRequest)
+
+ // w = httptest.NewRecorder()
+ // client.QueryRangeHandler(w, r)
+ // resp = w.Result()
+
+ // err = testStatusCodeMatch(resp.StatusCode, http.StatusBadRequest)
+ // if err != nil {
+ // t.Error(err)
+ // }
+
+ // err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "proxy-error"})
+ // if err != nil {
+ // t.Error(err)
+ // }
+
+}
+
+func TestDeltaProxyCacheRequestFastForwardUrlError(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("throw_ffurl_error=1&step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ oc.FastForwardDisable = false
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"ffstatus": "err"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestWithRefresh(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ r.Header.Set(headers.NameCacheControl, headers.ValueNoCache)
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "purge"})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestDeltaProxyCacheRequestWithRefreshError(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ r.Header.Set(headers.NameCacheControl, headers.ValueNoCache)
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadRequest)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadRequest)
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestWithUnmarshalAndUpstreamErrors(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test" // disable direct-memory and force marshaling
+
+ client.RangeCacheKey = "testkey"
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ key := oc.Host + ".409d551e3653f5ad5aa9acbdac8d4ac3"
+
+ _, _, err = client.cache.Retrieve(key, false)
+ if err != nil {
+ t.Error(err)
+ }
+
+ client.cache.Store(key, []byte("foo"), time.Duration(30)*time.Second)
+
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadRequest)
+ client.cache.Store(key, []byte("foo"), time.Duration(30)*time.Second)
+
+ r.URL = u
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadRequest)
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequest_BadParams(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ const query = "some_query_here{}"
+ step := time.Duration(300) * time.Second
+ end := time.Now()
+ start := end.Add(-time.Duration(6) * time.Hour)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ // Intentional typo &q instead of &query to force a proxied request due to ParseTimeRangeQuery() error
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&q=%s", int(step.Seconds()), start.Unix(), end.Unix(), query)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadRequest)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // ensure the request was sent through the proxy instead of the DeltaProxyCache
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestCacheMissUnmarshalFailed(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test" // disable direct-memory and force marshaling
+
+ oc.FastForwardDisable = true
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadRequest)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ _, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadRequest)
+ if err != nil {
+ t.Error(err)
+ }
+
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadPayload)
+ r.URL = u
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ body, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ _, err = client.UnmarshalTimeseries(body)
+ if err == nil {
+ t.Errorf("expected unmarshaling error for %s", string(body))
+ }
+
+}
+
+func TestDeltaProxyCacheRequestOutOfWindow(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+
+ oc.FastForwardDisable = true
+
+ query := "some_query_here{}"
+ step := time.Duration(300) * time.Second
+ // Times are out-of-window for being cacheable
+ start := time.Unix(0, 0)
+ end := time.Unix(1800, 0)
+
+ // we still expect the same results
+ expected, _, _ := promsim.GetTimeSeriesData(query, start, end, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), start.Unix(), end.Unix(), query)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // Fully Out-of-Window Requests should be proxied and not cached
+ testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+
+ // do it again to ensure another cache miss
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ // Fully Out-of-Window Requests should be proxied and not cached
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestDeltaProxyCacheRequestBadGateway(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+ rsc.CacheConfig.CacheType = "test"
+
+ oc.FastForwardDisable = true
+
+ r.Header.Set(headers.NameCacheControl, headers.ValueNoCache)
+
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsBadGateway)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadGateway)
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequest_BackfillTolerance(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+
+ oc.BackfillTolerance = time.Duration(300) * time.Second
+ oc.FastForwardDisable = true
+
+ query := "some_query_here{}"
+ step := time.Duration(300) * time.Second
+
+ now := time.Now()
+ x := timeseries.Extent{Start: now.Add(-time.Duration(6) * time.Hour), End: now}
+ xn := timeseries.Extent{Start: now.Add(-time.Duration(6) * time.Hour).Truncate(step), End: now.Truncate(step)}
+
+ // We can predict what slice will need to be fetched and ensure that is only what is requested upstream
+ expectedFetched := fmt.Sprintf("[%d:%d]", xn.End.Unix(), xn.End.Unix())
+ expected, _, _ := promsim.GetTimeSeriesData(query, xn.Start, xn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), x.Start.Unix(), x.End.Unix(), query)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ // get cache partial hit coverage too by repeating:
+ w = httptest.NewRecorder()
+ client.QueryRangeHandler(w, r)
+ resp = w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "phit"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"fetched": expectedFetched})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDeltaProxyCacheRequestFFTTLBiggerThanStep(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessDPC()
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ client := rsc.OriginClient.(*TestClient)
+ oc := rsc.OriginConfig
+
+ oc.FastForwardDisable = false
+
+ step := time.Duration(300) * time.Second
+ oc.FastForwardTTL = step + 1
+
+ now := time.Now()
+ end := now.Add(-time.Duration(12) * time.Hour)
+
+ extr := timeseries.Extent{Start: end.Add(-time.Duration(18) * time.Hour), End: end}
+ extn := timeseries.Extent{Start: extr.Start.Truncate(step), End: extr.End.Truncate(step)}
+
+ expected, _, _ := promsim.GetTimeSeriesData(queryReturnsOKNoLatency, extn.Start, extn.End, step)
+
+ u := r.URL
+ u.Path = "/api/v1/query_range"
+ u.RawQuery = fmt.Sprintf("step=%d&start=%d&end=%d&query=%s", int(step.Seconds()), extr.Start.Unix(), extr.End.Unix(), queryReturnsOKNoLatency)
+
+ client.QueryRangeHandler(w, r)
+ resp := w.Result()
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), expected)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"status": "kmiss"})
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"ffstatus": "off"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
diff --git a/internal/proxy/engines/document.go b/internal/proxy/engines/document.go
new file mode 100644
index 000000000..2547a6f7c
--- /dev/null
+++ b/internal/proxy/engines/document.go
@@ -0,0 +1,203 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "errors"
+ "io/ioutil"
+ "net/http"
+ "strconv"
+ "strings"
+
+ txe "github.com/Comcast/trickster/internal/proxy/errors"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/internal/util/log"
+)
+
+//go:generate msgp
+
+// HTTPDocument represents a full HTTP Response/Cache Document with unbuffered body
+type HTTPDocument struct {
+ StatusCode int `msg:"status_code"`
+ Status string `msg:"status"`
+ Headers map[string][]string `msg:"headers"`
+ Body []byte `msg:"body"`
+ ContentLength int64 `msg:"content_length"`
+ ContentType string `msg:"content_type"`
+ CachingPolicy *CachingPolicy `msg:"caching_policy"`
+ // Ranges is the list of Byte Ranges contained in the body of this document
+ Ranges byterange.Ranges `msg:"ranges"`
+ RangeParts byterange.MultipartByteRanges `msg:"-"`
+ // StoredRangeParts is a version of RangeParts that can be exported to MessagePack
+ StoredRangeParts map[string]*byterange.MultipartByteRange `msg:"range_parts"`
+
+ rangePartsLoaded bool
+ isFulfillment bool
+ isLoaded bool
+ timeseries timeseries.Timeseries
+ isCompressed bool
+}
+
+// Size returns the size of the HTTPDocument's headers, CachingPolicy, RangeParts, Body and timeseries data
+func (d *HTTPDocument) Size() int {
+ var i int
+ i += len(headers.String(http.Header(d.Headers)))
+ i += len(d.Body)
+ if d.RangeParts != nil {
+ for _, p := range d.RangeParts {
+ i += p.Msgsize()
+ }
+ }
+ if d.CachingPolicy != nil {
+ i += d.CachingPolicy.Msgsize()
+ }
+ if d.timeseries != nil {
+ i += d.timeseries.Size()
+ }
+ return i
+}
+
+// SetBody sets the Document Body as well as the Content Length, based on the length of body.
+// This assumes that the caller has checked that the request is not a Range request
+func (d *HTTPDocument) SetBody(body []byte) {
+ if body == nil {
+ return
+ }
+ d.Body = body
+ bl := int64(len(d.Body))
+ if d.ContentLength == -1 || d.ContentLength != bl {
+ d.ContentLength = bl
+ }
+ if d.Headers == nil {
+ d.Headers = make(http.Header)
+ }
+ http.Header(d.Headers).Set(headers.NameContentLength, strconv.Itoa(len(body)))
+}
+
+// LoadRangeParts convert a StoredRangeParts into a RangeParts
+func (d *HTTPDocument) LoadRangeParts() {
+
+ if d.rangePartsLoaded {
+ return
+ }
+
+ if d.StoredRangeParts != nil && len(d.StoredRangeParts) > 0 {
+ d.RangeParts = make(byterange.MultipartByteRanges)
+ for _, p := range d.StoredRangeParts {
+ d.RangeParts[p.Range] = p
+ }
+ d.Ranges = d.RangeParts.Ranges()
+ }
+ d.rangePartsLoaded = true
+}
+
+// ParsePartialContentBody parses a Partial Content response body into 0 or more discrete parts
+func (d *HTTPDocument) ParsePartialContentBody(resp *http.Response, body []byte) {
+
+ ct := resp.Header.Get(headers.NameContentType)
+ if cr := resp.Header.Get(headers.NameContentRange); cr != "" {
+ if !strings.HasPrefix(ct, headers.ValueMultipartByteRanges) {
+ d.ContentType = ct
+ }
+ r, cl, err := byterange.ParseContentRangeHeader(cr)
+ d.ContentLength = int64(cl)
+ if err == nil && (r.Start >= 0 || r.End >= 0) {
+ mpbr := &byterange.MultipartByteRange{Range: r, Content: body}
+ if d.RangeParts == nil {
+ d.RangeParts = byterange.MultipartByteRanges{r: mpbr}
+ } else {
+ d.RangeParts[r] = mpbr
+ }
+ }
+ if d.RangeParts != nil {
+ byterange.MultipartByteRanges(d.RangeParts).Compress()
+ d.Ranges = d.RangeParts.Ranges()
+
+ if d.RangeParts != nil {
+ d.StoredRangeParts = d.RangeParts.PackableMultipartByteRanges()
+ }
+ }
+ } else if strings.HasPrefix(ct, headers.ValueMultipartByteRanges) {
+ p, ct, r, cl, err := byterange.ParseMultipartRangeResponseBody(ioutil.NopCloser(bytes.NewBuffer(body)), ct)
+ if err == nil {
+ if d.RangeParts == nil {
+ d.Ranges = r
+ d.RangeParts = p
+ } else {
+ d.RangeParts.Merge(p)
+ d.Ranges = d.RangeParts.Ranges()
+ }
+ d.StoredRangeParts = d.RangeParts.PackableMultipartByteRanges()
+ d.ContentLength = int64(cl)
+ if !strings.HasPrefix(ct, headers.ValueMultipartByteRanges) {
+ d.ContentType = ct
+ }
+ d.RangeParts.Compress()
+ d.Ranges = d.RangeParts.Ranges()
+ } else {
+ log.Error("unable to parse multipart range response body", log.Pairs{"detail": err.Error})
+ }
+ } else {
+ if !strings.HasPrefix(ct, headers.ValueMultipartByteRanges) {
+ d.ContentType = ct
+ }
+ d.SetBody(body)
+ }
+
+ if d.ContentLength > 0 && len(d.RangeParts) == 1 &&
+ d.RangeParts[d.RangeParts.Ranges()[0]].Range.Start == 0 &&
+ d.RangeParts[d.RangeParts.Ranges()[0]].Range.End == d.ContentLength-1 {
+ d.FulfillContentBody()
+ }
+
+ http.Header(d.Headers).Del(headers.NameContentType)
+
+}
+
+// FulfillContentBody will concatenate the document's Range parts into a single, full content body
+// the caller must know that document's multipart ranges include full content length before calling this method
+func (d *HTTPDocument) FulfillContentBody() error {
+
+ if d.RangeParts == nil || len(d.RangeParts) == 0 {
+ d.SetBody(nil)
+ return txe.ErrNoRanges
+ }
+
+ d.RangeParts.Compress()
+ d.Ranges = d.RangeParts.Ranges()
+
+ if len(d.RangeParts) != 1 {
+ d.SetBody(nil)
+ return errors.New("cached parts do not comprise the full body")
+ }
+
+ p := d.RangeParts[d.Ranges[0]]
+ r := p.Range
+
+ if r.Start != 0 || r.End != d.ContentLength-1 {
+ d.SetBody(nil)
+ return errors.New("cached parts do not comprise the full body")
+ }
+
+ d.StatusCode = http.StatusOK
+
+ d.Ranges = nil
+ d.RangeParts = nil
+ d.StoredRangeParts = nil
+ d.SetBody(p.Content)
+ return nil
+}
diff --git a/internal/proxy/engines/document_gen.go b/internal/proxy/engines/document_gen.go
new file mode 100644
index 000000000..44c752728
--- /dev/null
+++ b/internal/proxy/engines/document_gen.go
@@ -0,0 +1,529 @@
+package engines
+
+// Code generated by github.com/tinylib/msgp DO NOT EDIT.
+
+import (
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+ "github.com/tinylib/msgp/msgp"
+)
+
+// DecodeMsg implements msgp.Decodable
+func (z *HTTPDocument) DecodeMsg(dc *msgp.Reader) (err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, err = dc.ReadMapKeyPtr()
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "status_code":
+ z.StatusCode, err = dc.ReadInt()
+ if err != nil {
+ return
+ }
+ case "status":
+ z.Status, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ case "headers":
+ var zb0002 uint32
+ zb0002, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ if z.Headers == nil {
+ z.Headers = make(map[string][]string, zb0002)
+ } else if len(z.Headers) > 0 {
+ for key := range z.Headers {
+ delete(z.Headers, key)
+ }
+ }
+ for zb0002 > 0 {
+ zb0002--
+ var za0001 string
+ var za0002 []string
+ za0001, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ var zb0003 uint32
+ zb0003, err = dc.ReadArrayHeader()
+ if err != nil {
+ return
+ }
+ if cap(za0002) >= int(zb0003) {
+ za0002 = (za0002)[:zb0003]
+ } else {
+ za0002 = make([]string, zb0003)
+ }
+ for za0003 := range za0002 {
+ za0002[za0003], err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ }
+ z.Headers[za0001] = za0002
+ }
+ case "body":
+ z.Body, err = dc.ReadBytes(z.Body)
+ if err != nil {
+ return
+ }
+ case "content_length":
+ z.ContentLength, err = dc.ReadInt64()
+ if err != nil {
+ return
+ }
+ case "content_type":
+ z.ContentType, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ case "caching_policy":
+ if dc.IsNil() {
+ err = dc.ReadNil()
+ if err != nil {
+ return
+ }
+ z.CachingPolicy = nil
+ } else {
+ if z.CachingPolicy == nil {
+ z.CachingPolicy = new(CachingPolicy)
+ }
+ err = z.CachingPolicy.DecodeMsg(dc)
+ if err != nil {
+ return
+ }
+ }
+ case "ranges":
+ err = z.Ranges.DecodeMsg(dc)
+ if err != nil {
+ return
+ }
+ case "range_parts":
+ var zb0004 uint32
+ zb0004, err = dc.ReadMapHeader()
+ if err != nil {
+ return
+ }
+ if z.StoredRangeParts == nil {
+ z.StoredRangeParts = make(map[string]*byterange.MultipartByteRange, zb0004)
+ } else if len(z.StoredRangeParts) > 0 {
+ for key := range z.StoredRangeParts {
+ delete(z.StoredRangeParts, key)
+ }
+ }
+ for zb0004 > 0 {
+ zb0004--
+ var za0004 string
+ var za0005 *byterange.MultipartByteRange
+ za0004, err = dc.ReadString()
+ if err != nil {
+ return
+ }
+ if dc.IsNil() {
+ err = dc.ReadNil()
+ if err != nil {
+ return
+ }
+ za0005 = nil
+ } else {
+ if za0005 == nil {
+ za0005 = new(byterange.MultipartByteRange)
+ }
+ err = za0005.DecodeMsg(dc)
+ if err != nil {
+ return
+ }
+ }
+ z.StoredRangeParts[za0004] = za0005
+ }
+ default:
+ err = dc.Skip()
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// EncodeMsg implements msgp.Encodable
+func (z *HTTPDocument) EncodeMsg(en *msgp.Writer) (err error) {
+ // map header, size 9
+ // write "status_code"
+ err = en.Append(0x89, 0xab, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x5f, 0x63, 0x6f, 0x64, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteInt(z.StatusCode)
+ if err != nil {
+ return
+ }
+ // write "status"
+ err = en.Append(0xa6, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73)
+ if err != nil {
+ return
+ }
+ err = en.WriteString(z.Status)
+ if err != nil {
+ return
+ }
+ // write "headers"
+ err = en.Append(0xa7, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x73)
+ if err != nil {
+ return
+ }
+ err = en.WriteMapHeader(uint32(len(z.Headers)))
+ if err != nil {
+ return
+ }
+ for za0001, za0002 := range z.Headers {
+ err = en.WriteString(za0001)
+ if err != nil {
+ return
+ }
+ err = en.WriteArrayHeader(uint32(len(za0002)))
+ if err != nil {
+ return
+ }
+ for za0003 := range za0002 {
+ err = en.WriteString(za0002[za0003])
+ if err != nil {
+ return
+ }
+ }
+ }
+ // write "body"
+ err = en.Append(0xa4, 0x62, 0x6f, 0x64, 0x79)
+ if err != nil {
+ return
+ }
+ err = en.WriteBytes(z.Body)
+ if err != nil {
+ return
+ }
+ // write "content_length"
+ err = en.Append(0xae, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x5f, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68)
+ if err != nil {
+ return
+ }
+ err = en.WriteInt64(z.ContentLength)
+ if err != nil {
+ return
+ }
+ // write "content_type"
+ err = en.Append(0xac, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x5f, 0x74, 0x79, 0x70, 0x65)
+ if err != nil {
+ return
+ }
+ err = en.WriteString(z.ContentType)
+ if err != nil {
+ return
+ }
+ // write "caching_policy"
+ err = en.Append(0xae, 0x63, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x67, 0x5f, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79)
+ if err != nil {
+ return
+ }
+ if z.CachingPolicy == nil {
+ err = en.WriteNil()
+ if err != nil {
+ return
+ }
+ } else {
+ err = z.CachingPolicy.EncodeMsg(en)
+ if err != nil {
+ return
+ }
+ }
+ // write "ranges"
+ err = en.Append(0xa6, 0x72, 0x61, 0x6e, 0x67, 0x65, 0x73)
+ if err != nil {
+ return
+ }
+ err = z.Ranges.EncodeMsg(en)
+ if err != nil {
+ return
+ }
+ // write "range_parts"
+ err = en.Append(0xab, 0x72, 0x61, 0x6e, 0x67, 0x65, 0x5f, 0x70, 0x61, 0x72, 0x74, 0x73)
+ if err != nil {
+ return
+ }
+ err = en.WriteMapHeader(uint32(len(z.StoredRangeParts)))
+ if err != nil {
+ return
+ }
+ for za0004, za0005 := range z.StoredRangeParts {
+ err = en.WriteString(za0004)
+ if err != nil {
+ return
+ }
+ if za0005 == nil {
+ err = en.WriteNil()
+ if err != nil {
+ return
+ }
+ } else {
+ err = za0005.EncodeMsg(en)
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// MarshalMsg implements msgp.Marshaler
+func (z *HTTPDocument) MarshalMsg(b []byte) (o []byte, err error) {
+ o = msgp.Require(b, z.Msgsize())
+ // map header, size 9
+ // string "status_code"
+ o = append(o, 0x89, 0xab, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x5f, 0x63, 0x6f, 0x64, 0x65)
+ o = msgp.AppendInt(o, z.StatusCode)
+ // string "status"
+ o = append(o, 0xa6, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73)
+ o = msgp.AppendString(o, z.Status)
+ // string "headers"
+ o = append(o, 0xa7, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x73)
+ o = msgp.AppendMapHeader(o, uint32(len(z.Headers)))
+ for za0001, za0002 := range z.Headers {
+ o = msgp.AppendString(o, za0001)
+ o = msgp.AppendArrayHeader(o, uint32(len(za0002)))
+ for za0003 := range za0002 {
+ o = msgp.AppendString(o, za0002[za0003])
+ }
+ }
+ // string "body"
+ o = append(o, 0xa4, 0x62, 0x6f, 0x64, 0x79)
+ o = msgp.AppendBytes(o, z.Body)
+ // string "content_length"
+ o = append(o, 0xae, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x5f, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68)
+ o = msgp.AppendInt64(o, z.ContentLength)
+ // string "content_type"
+ o = append(o, 0xac, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x5f, 0x74, 0x79, 0x70, 0x65)
+ o = msgp.AppendString(o, z.ContentType)
+ // string "caching_policy"
+ o = append(o, 0xae, 0x63, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x67, 0x5f, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79)
+ if z.CachingPolicy == nil {
+ o = msgp.AppendNil(o)
+ } else {
+ o, err = z.CachingPolicy.MarshalMsg(o)
+ if err != nil {
+ return
+ }
+ }
+ // string "ranges"
+ o = append(o, 0xa6, 0x72, 0x61, 0x6e, 0x67, 0x65, 0x73)
+ o, err = z.Ranges.MarshalMsg(o)
+ if err != nil {
+ return
+ }
+ // string "range_parts"
+ o = append(o, 0xab, 0x72, 0x61, 0x6e, 0x67, 0x65, 0x5f, 0x70, 0x61, 0x72, 0x74, 0x73)
+ o = msgp.AppendMapHeader(o, uint32(len(z.StoredRangeParts)))
+ for za0004, za0005 := range z.StoredRangeParts {
+ o = msgp.AppendString(o, za0004)
+ if za0005 == nil {
+ o = msgp.AppendNil(o)
+ } else {
+ o, err = za0005.MarshalMsg(o)
+ if err != nil {
+ return
+ }
+ }
+ }
+ return
+}
+
+// UnmarshalMsg implements msgp.Unmarshaler
+func (z *HTTPDocument) UnmarshalMsg(bts []byte) (o []byte, err error) {
+ var field []byte
+ _ = field
+ var zb0001 uint32
+ zb0001, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ for zb0001 > 0 {
+ zb0001--
+ field, bts, err = msgp.ReadMapKeyZC(bts)
+ if err != nil {
+ return
+ }
+ switch msgp.UnsafeString(field) {
+ case "status_code":
+ z.StatusCode, bts, err = msgp.ReadIntBytes(bts)
+ if err != nil {
+ return
+ }
+ case "status":
+ z.Status, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ case "headers":
+ var zb0002 uint32
+ zb0002, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ if z.Headers == nil {
+ z.Headers = make(map[string][]string, zb0002)
+ } else if len(z.Headers) > 0 {
+ for key := range z.Headers {
+ delete(z.Headers, key)
+ }
+ }
+ for zb0002 > 0 {
+ var za0001 string
+ var za0002 []string
+ zb0002--
+ za0001, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ var zb0003 uint32
+ zb0003, bts, err = msgp.ReadArrayHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ if cap(za0002) >= int(zb0003) {
+ za0002 = (za0002)[:zb0003]
+ } else {
+ za0002 = make([]string, zb0003)
+ }
+ for za0003 := range za0002 {
+ za0002[za0003], bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ }
+ z.Headers[za0001] = za0002
+ }
+ case "body":
+ z.Body, bts, err = msgp.ReadBytesBytes(bts, z.Body)
+ if err != nil {
+ return
+ }
+ case "content_length":
+ z.ContentLength, bts, err = msgp.ReadInt64Bytes(bts)
+ if err != nil {
+ return
+ }
+ case "content_type":
+ z.ContentType, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ case "caching_policy":
+ if msgp.IsNil(bts) {
+ bts, err = msgp.ReadNilBytes(bts)
+ if err != nil {
+ return
+ }
+ z.CachingPolicy = nil
+ } else {
+ if z.CachingPolicy == nil {
+ z.CachingPolicy = new(CachingPolicy)
+ }
+ bts, err = z.CachingPolicy.UnmarshalMsg(bts)
+ if err != nil {
+ return
+ }
+ }
+ case "ranges":
+ bts, err = z.Ranges.UnmarshalMsg(bts)
+ if err != nil {
+ return
+ }
+ case "range_parts":
+ var zb0004 uint32
+ zb0004, bts, err = msgp.ReadMapHeaderBytes(bts)
+ if err != nil {
+ return
+ }
+ if z.StoredRangeParts == nil {
+ z.StoredRangeParts = make(map[string]*byterange.MultipartByteRange, zb0004)
+ } else if len(z.StoredRangeParts) > 0 {
+ for key := range z.StoredRangeParts {
+ delete(z.StoredRangeParts, key)
+ }
+ }
+ for zb0004 > 0 {
+ var za0004 string
+ var za0005 *byterange.MultipartByteRange
+ zb0004--
+ za0004, bts, err = msgp.ReadStringBytes(bts)
+ if err != nil {
+ return
+ }
+ if msgp.IsNil(bts) {
+ bts, err = msgp.ReadNilBytes(bts)
+ if err != nil {
+ return
+ }
+ za0005 = nil
+ } else {
+ if za0005 == nil {
+ za0005 = new(byterange.MultipartByteRange)
+ }
+ bts, err = za0005.UnmarshalMsg(bts)
+ if err != nil {
+ return
+ }
+ }
+ z.StoredRangeParts[za0004] = za0005
+ }
+ default:
+ bts, err = msgp.Skip(bts)
+ if err != nil {
+ return
+ }
+ }
+ }
+ o = bts
+ return
+}
+
+// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
+func (z *HTTPDocument) Msgsize() (s int) {
+ s = 1 + 12 + msgp.IntSize + 7 + msgp.StringPrefixSize + len(z.Status) + 8 + msgp.MapHeaderSize
+ if z.Headers != nil {
+ for za0001, za0002 := range z.Headers {
+ _ = za0002
+ s += msgp.StringPrefixSize + len(za0001) + msgp.ArrayHeaderSize
+ for za0003 := range za0002 {
+ s += msgp.StringPrefixSize + len(za0002[za0003])
+ }
+ }
+ }
+ s += 5 + msgp.BytesPrefixSize + len(z.Body) + 15 + msgp.Int64Size + 13 + msgp.StringPrefixSize + len(z.ContentType) + 15
+ if z.CachingPolicy == nil {
+ s += msgp.NilSize
+ } else {
+ s += z.CachingPolicy.Msgsize()
+ }
+ s += 7 + z.Ranges.Msgsize() + 12 + msgp.MapHeaderSize
+ if z.StoredRangeParts != nil {
+ for za0004, za0005 := range z.StoredRangeParts {
+ _ = za0005
+ s += msgp.StringPrefixSize + len(za0004)
+ if za0005 == nil {
+ s += msgp.NilSize
+ } else {
+ s += za0005.Msgsize()
+ }
+ }
+ }
+ return
+}
diff --git a/internal/proxy/engines/document_gen_test.go b/internal/proxy/engines/document_gen_test.go
new file mode 100644
index 000000000..563d6ccfa
--- /dev/null
+++ b/internal/proxy/engines/document_gen_test.go
@@ -0,0 +1,123 @@
+package engines
+
+// Code generated by github.com/tinylib/msgp DO NOT EDIT.
+
+import (
+ "bytes"
+ "testing"
+
+ "github.com/tinylib/msgp/msgp"
+)
+
+func TestMarshalUnmarshalHTTPDocument(t *testing.T) {
+ v := HTTPDocument{}
+ bts, err := v.MarshalMsg(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ left, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left)
+ }
+
+ left, err = msgp.Skip(bts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) > 0 {
+ t.Errorf("%d bytes left over after Skip(): %q", len(left), left)
+ }
+}
+
+func BenchmarkMarshalMsgHTTPDocument(b *testing.B) {
+ v := HTTPDocument{}
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.MarshalMsg(nil)
+ }
+}
+
+func BenchmarkAppendMsgHTTPDocument(b *testing.B) {
+ v := HTTPDocument{}
+ bts := make([]byte, 0, v.Msgsize())
+ bts, _ = v.MarshalMsg(bts[0:0])
+ b.SetBytes(int64(len(bts)))
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ bts, _ = v.MarshalMsg(bts[0:0])
+ }
+}
+
+func BenchmarkUnmarshalHTTPDocument(b *testing.B) {
+ v := HTTPDocument{}
+ bts, _ := v.MarshalMsg(nil)
+ b.ReportAllocs()
+ b.SetBytes(int64(len(bts)))
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ _, err := v.UnmarshalMsg(bts)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
+
+func TestEncodeDecodeHTTPDocument(t *testing.T) {
+ v := HTTPDocument{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+
+ m := v.Msgsize()
+ if buf.Len() > m {
+ t.Logf("WARNING: Msgsize() for %v is inaccurate", v)
+ }
+
+ vn := HTTPDocument{}
+ err := msgp.Decode(&buf, &vn)
+ if err != nil {
+ t.Error(err)
+ }
+
+ buf.Reset()
+ msgp.Encode(&buf, &v)
+ err = msgp.NewReader(&buf).Skip()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func BenchmarkEncodeHTTPDocument(b *testing.B) {
+ v := HTTPDocument{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ en := msgp.NewWriter(msgp.Nowhere)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ v.EncodeMsg(en)
+ }
+ en.Flush()
+}
+
+func BenchmarkDecodeHTTPDocument(b *testing.B) {
+ v := HTTPDocument{}
+ var buf bytes.Buffer
+ msgp.Encode(&buf, &v)
+ b.SetBytes(int64(buf.Len()))
+ rd := msgp.NewEndlessReader(buf.Bytes(), b)
+ dc := msgp.NewReader(rd)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ err := v.DecodeMsg(dc)
+ if err != nil {
+ b.Fatal(err)
+ }
+ }
+}
diff --git a/internal/proxy/engines/document_test.go b/internal/proxy/engines/document_test.go
new file mode 100644
index 000000000..f81b2d4e5
--- /dev/null
+++ b/internal/proxy/engines/document_test.go
@@ -0,0 +1,225 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "net/http"
+ "strings"
+ "testing"
+
+ txe "github.com/Comcast/trickster/internal/proxy/errors"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+)
+
+func TestDocumentFromHTTPResponse(t *testing.T) {
+
+ expected := []byte("1234")
+
+ resp := &http.Response{}
+ resp.Header = http.Header{headers.NameContentRange: []string{"bytes 1-4/8"}}
+ resp.StatusCode = 206
+ d := DocumentFromHTTPResponse(resp, []byte("1234"), nil)
+
+ if len(d.Ranges) != 1 {
+ t.Errorf("expected 1 got %d", len(d.Ranges))
+ } else if string(d.RangeParts[d.Ranges[0]].Content) != string(expected) {
+ t.Errorf("expected %s got %s", string(expected), string(d.Body))
+ }
+
+ if d.StatusCode != 206 {
+ t.Errorf("expected %d got %d", 206, d.StatusCode)
+ }
+
+}
+
+func TestCachingPolicyString(t *testing.T) {
+
+ cp := &CachingPolicy{NoTransform: true}
+ s := cp.String()
+
+ i := strings.Index(s, `"no_transform":true`)
+ if i < 1 {
+ t.Errorf("expected value > 1, got %d", i)
+ }
+
+}
+
+func TestSetBody(t *testing.T) {
+
+ r := byterange.Range{Start: 0, End: 10}
+ d := &HTTPDocument{ContentLength: -1, RangeParts: byterange.MultipartByteRanges{r: &byterange.MultipartByteRange{Range: r, Content: []byte("01234567890")}}}
+ d.SetBody([]byte("testing"))
+
+ if d.ContentLength < 0 {
+ t.Errorf("expected value > 0, got %d", d.ContentLength)
+ }
+}
+
+func TestSize(t *testing.T) {
+ r := byterange.Range{Start: 0, End: 10}
+ d := &HTTPDocument{ContentLength: -1, RangeParts: byterange.MultipartByteRanges{r: &byterange.MultipartByteRange{Range: r, Content: []byte("01234567890")}}}
+
+ i := d.Size()
+
+ if i != 62 {
+ t.Errorf("expected %d got %d", 62, i)
+ }
+
+}
+
+func TestFulfillContentBody(t *testing.T) {
+ d := &HTTPDocument{}
+ err := d.FulfillContentBody()
+ if err != txe.ErrNoRanges {
+ if err != nil {
+ t.Error(err)
+ } else {
+ t.Errorf("expected error: %s", txe.ErrNoRanges.Error())
+ }
+ }
+}
+
+func TestParsePartialContentBodyNoRanges(t *testing.T) {
+
+ d := &HTTPDocument{}
+ resp := &http.Response{Header: make(http.Header)}
+ d.ParsePartialContentBody(resp, []byte("test"))
+
+ if string(d.Body) != "test" {
+ t.Errorf("expected %s got %s", "test", string(d.Body))
+ }
+
+}
+
+func TestParsePartialContentBodySingleRange(t *testing.T) {
+ d := &HTTPDocument{}
+ d.Ranges = make(byterange.Ranges, 0)
+ d.RangeParts = make(byterange.MultipartByteRanges)
+ d.StoredRangeParts = make(map[string]*byterange.MultipartByteRange)
+
+ resp := &http.Response{Header: http.Header{
+ headers.NameContentRange: []string{"bytes 0-10/1222"},
+ }}
+
+ d.ParsePartialContentBody(resp, []byte("Lorem ipsum"))
+
+ if string(d.Body) != "" {
+ t.Errorf("expected %s got %s", "", string(d.Body))
+ }
+
+ if len(d.RangeParts) != 1 {
+ t.Errorf("expected %d got %d", 1, len(d.RangeParts))
+ }
+}
+
+func TestParsePartialContentBodyMultipart(t *testing.T) {
+ d := &HTTPDocument{}
+ d.Ranges = make(byterange.Ranges, 0)
+ d.RangeParts = make(byterange.MultipartByteRanges)
+ d.StoredRangeParts = make(map[string]*byterange.MultipartByteRange)
+
+ resp := &http.Response{
+ StatusCode: http.StatusPartialContent,
+ Header: http.Header{},
+ }
+
+ resp.Header.Set(headers.NameContentType, "multipart/byteranges; boundary=c4fb8e6049a6fdb126d32fa0b15c21e3")
+ resp.Header.Set(headers.NameContentLength, "271")
+
+ d.ParsePartialContentBody(resp, []byte(`
+--c4fb8e6049a6fdb126d32fa0b15c21e3
+Content-Range: bytes 0-6/1222
+Content-Type: text/plain; charset=utf-8
+
+Lorem i
+--c4fb8e6049a6fdb126d32fa0b15c21e3
+Content-Range: bytes 10-20/1222
+Content-Type: text/plain; charset=utf-8
+
+m dolor sit
+--c4fb8e6049a6fdb126d32fa0b15c21e3--`))
+
+ if string(d.Body) != "" {
+ t.Errorf("expected %s got %s", "", string(d.Body))
+ }
+
+ if len(d.RangeParts) != 2 {
+ t.Errorf("expected %d got %d", 2, len(d.RangeParts))
+ }
+}
+
+func TestParsePartialContentBodyMultipartBadBody(t *testing.T) {
+ d := &HTTPDocument{}
+ d.Ranges = make(byterange.Ranges, 0)
+ d.RangeParts = make(byterange.MultipartByteRanges)
+ d.StoredRangeParts = make(map[string]*byterange.MultipartByteRange)
+
+ resp := &http.Response{
+ StatusCode: http.StatusPartialContent,
+ Header: http.Header{},
+ }
+
+ resp.Header.Set(headers.NameContentType, "multipart/byteranges; boundary=c4fb8e6049a6fdb126d32fa0b15c21e3")
+ resp.Header.Set(headers.NameContentLength, "271")
+
+ d.ParsePartialContentBody(resp, []byte(`
+--c4fb8e6049a6fdb126d32fa0b15c21e3
+Content-Range: bytes 0-6/1222
+Content-Type: text/plain; charset=utf-8
+
+Lorem i
+--c4fb8e6049a6fdb126d32fa0b15c21e3
+Content-Range: baytes 1s0-20/12s22x
+Content-Type: text/plain; charset=utf-8
+
+m dolor sit
+--c4fb8e6049a6fdb126d32fa0b15c21e3--`))
+
+ if string(d.Body) != "" {
+ t.Errorf("expected %s got %s", "", string(d.Body))
+ }
+
+ if len(d.RangeParts) != 0 {
+ t.Errorf("expected %d got %d", 0, len(d.RangeParts))
+ }
+
+}
+
+func TestLoadRangeParts(t *testing.T) {
+
+ d := &HTTPDocument{
+ rangePartsLoaded: true,
+ StoredRangeParts: map[string]*byterange.MultipartByteRange{
+ "range": {
+ Range: byterange.Range{Start: 0, End: 8},
+ Content: []byte("trickster"),
+ },
+ },
+ }
+
+ // test the short circuit
+ d.LoadRangeParts()
+ if d.Ranges != nil {
+ t.Errorf("expected nil got %s", d.Ranges.String())
+ }
+
+ // and now the main functionality
+ d.rangePartsLoaded = false
+ d.LoadRangeParts()
+ if len(d.Ranges) != 1 {
+ t.Errorf("expected %d got %d", 1, len(d.Ranges))
+ }
+
+}
diff --git a/internal/proxy/engines/httpproxy.go b/internal/proxy/engines/httpproxy.go
new file mode 100644
index 000000000..9835ba655
--- /dev/null
+++ b/internal/proxy/engines/httpproxy.go
@@ -0,0 +1,217 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "io"
+ "io/ioutil"
+ "math"
+ "net/http"
+ "strconv"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/params"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+// Reqs is for Progressive Collapsed Forwarding
+var Reqs sync.Map
+
+// HTTPBlockSize represents 32K of bytes
+const HTTPBlockSize = 32 * 1024
+
+// DoProxy proxies an inbound request to its corresponding upstream origin with no caching features
+func DoProxy(w io.Writer, r *http.Request) *http.Response {
+
+ rsc := request.GetResources(r)
+ pc := rsc.PathConfig
+ oc := rsc.OriginConfig
+
+ start := time.Now()
+ var elapsed time.Duration
+ var cacheStatusCode status.LookupStatus
+ var resp *http.Response
+ var reader io.Reader
+ if pc == nil || pc.CollapsedForwardingType != config.CFTypeProgressive {
+ reader, resp, _ = PrepareFetchReader(r)
+ cacheStatusCode = setStatusHeader(resp.StatusCode, resp.Header)
+ writer := PrepareResponseWriter(w, resp.StatusCode, resp.Header)
+ if writer != nil && reader != nil {
+ io.Copy(writer, reader)
+ }
+ } else {
+ pr := newProxyRequest(r, w)
+ key := oc.CacheKeyPrefix + "." + pr.DeriveCacheKey(nil, "")
+ result, ok := Reqs.Load(key)
+ if !ok {
+ var contentLength int64
+ reader, resp, contentLength = PrepareFetchReader(r)
+ cacheStatusCode = setStatusHeader(resp.StatusCode, resp.Header)
+ writer := PrepareResponseWriter(w, resp.StatusCode, resp.Header)
+ // Check if we know the content length and if it is less than our max object size.
+ if contentLength != 0 && contentLength < int64(oc.MaxObjectSizeBytes) {
+ pcf := NewPCF(resp, contentLength)
+ Reqs.Store(key, pcf)
+ // Blocks until server completes
+ go func() {
+ io.Copy(pcf, reader)
+ pcf.Close()
+ Reqs.Delete(key)
+ }()
+ pcf.AddClient(writer)
+ }
+ } else {
+ pcf, _ := result.(ProgressiveCollapseForwarder)
+ resp = pcf.GetResp()
+ writer := PrepareResponseWriter(w, resp.StatusCode, resp.Header)
+ pcf.AddClient(writer)
+ }
+ }
+ elapsed = time.Since(start)
+ recordResults(r, "HTTPProxy", cacheStatusCode, resp.StatusCode, r.URL.Path, "", elapsed.Seconds(), nil, resp.Header)
+ return resp
+}
+
+// PrepareResponseWriter prepares a response and returns an io.Writer for the data to be written to.
+// Used in Respond.
+func PrepareResponseWriter(w io.Writer, code int, header http.Header) io.Writer {
+ if rw, ok := w.(http.ResponseWriter); ok {
+ h := rw.Header()
+ headers.Merge(h, header)
+ headers.AddResponseHeaders(h)
+ rw.WriteHeader(code)
+ return rw
+ }
+ return w
+}
+
+// PrepareFetchReader prepares an http response and returns io.ReadCloser to
+// provide the response data, the response object and the content length.
+// Used in Fetch.
+func PrepareFetchReader(r *http.Request) (io.ReadCloser, *http.Response, int64) {
+
+ rsc := request.GetResources(r)
+ pc := rsc.PathConfig
+ oc := rsc.OriginConfig
+
+ var rc io.ReadCloser
+
+ if r != nil && r.Header != nil {
+ headers.AddProxyHeaders(r.RemoteAddr, r.Header)
+ }
+
+ headers.RemoveClientHeaders(r.Header)
+
+ if pc != nil {
+ headers.UpdateHeaders(r.Header, pc.RequestHeaders)
+ params.UpdateParams(r.URL.Query(), pc.RequestParams)
+ }
+
+ r.RequestURI = ""
+ resp, err := oc.HTTPClient.Do(r)
+ if err != nil {
+ log.Error("error downloading url", log.Pairs{"url": r.URL.String(), "detail": err.Error()})
+ // if there is an err and the response is nil, the server could not be reached; make a 502 for the downstream response
+ if resp == nil {
+ resp = &http.Response{StatusCode: http.StatusBadGateway, Request: r, Header: make(http.Header)}
+ }
+ if pc != nil {
+ headers.UpdateHeaders(resp.Header, pc.ResponseHeaders)
+ }
+ return nil, resp, 0
+ }
+
+ originalLen := int64(-1)
+ if v, ok := resp.Header[headers.NameContentLength]; ok {
+ originalLen, err = strconv.ParseInt(strings.Join(v, ""), 10, 64)
+ if err != nil {
+ originalLen = -1
+ }
+ resp.ContentLength = int64(originalLen)
+ }
+ rc = resp.Body
+
+ // warn if the clock between trickster and the origin is off by more than 1 minute
+ if date := resp.Header.Get(headers.NameDate); date != "" {
+ d, err := http.ParseTime(date)
+ if err == nil {
+ if offset := time.Since(d); time.Duration(math.Abs(float64(offset))) > time.Minute {
+ log.WarnOnce("clockoffset."+oc.Name,
+ "clock offset between trickster host and origin is high and may cause data anomalies",
+ log.Pairs{
+ "originName": oc.Name,
+ "tricksterTime": strconv.FormatInt(d.Add(offset).Unix(), 10),
+ "originTime": strconv.FormatInt(d.Unix(), 10),
+ "offset": strconv.FormatInt(int64(offset.Seconds()), 10) + "s",
+ })
+ }
+ }
+ }
+
+ hasCustomResponseBody := false
+ resp.Header.Del(headers.NameContentLength)
+
+ if pc != nil {
+ headers.UpdateHeaders(resp.Header, pc.ResponseHeaders)
+ hasCustomResponseBody = pc.HasCustomResponseBody
+ }
+
+ if hasCustomResponseBody {
+ rc = ioutil.NopCloser(bytes.NewBuffer(pc.ResponseBodyBytes))
+ }
+
+ return rc, resp, originalLen
+}
+
+// Respond sends an HTTP Response down to the requesting client
+func Respond(w http.ResponseWriter, code int, header http.Header, body []byte) {
+ PrepareResponseWriter(w, code, header)
+ w.Write(body)
+}
+
+func setStatusHeader(httpStatus int, header http.Header) status.LookupStatus {
+ st := status.LookupStatusProxyOnly
+ if httpStatus >= http.StatusBadRequest {
+ st = status.LookupStatusProxyError
+ }
+ headers.SetResultsHeader(header, "HTTPProxy", st.String(), "", nil)
+ return st
+}
+
+func recordResults(r *http.Request, engine string, cacheStatus status.LookupStatus, statusCode int, path, ffStatus string, elapsed float64, extents timeseries.ExtentList, header http.Header) {
+
+ rsc := request.GetResources(r)
+ pc := rsc.PathConfig
+ oc := rsc.OriginConfig
+
+ status := cacheStatus.String()
+
+ if pc != nil && !pc.NoMetrics {
+ httpStatus := strconv.Itoa(statusCode)
+ metrics.ProxyRequestStatus.WithLabelValues(oc.Name, oc.OriginType, r.Method, status, httpStatus, path).Inc()
+ if elapsed > 0 {
+ metrics.ProxyRequestDuration.WithLabelValues(oc.Name, oc.OriginType, r.Method, status, httpStatus, path).Observe(elapsed)
+ }
+ }
+ headers.SetResultsHeader(header, engine, status, ffStatus, extents)
+}
diff --git a/internal/proxy/engines/httpproxy_test.go b/internal/proxy/engines/httpproxy_test.go
new file mode 100644
index 000000000..11be4e86d
--- /dev/null
+++ b/internal/proxy/engines/httpproxy_test.go
@@ -0,0 +1,298 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "io/ioutil"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/config"
+ tc "github.com/Comcast/trickster/internal/proxy/context"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/internal/util/metrics"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+)
+
+func init() {
+ metrics.Init()
+}
+
+func TestDoProxy(t *testing.T) {
+
+ es := tu.NewTestServer(http.StatusOK, "test", nil)
+ defer es.Close()
+
+ err := config.Load("trickster", "test", []string{"-origin-url", es.URL, "-origin-type", "test", "-log-level", "debug"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ oc := config.Origins["default"]
+ pc := &config.PathConfig{
+ Path: "/",
+ RequestHeaders: map[string]string{},
+ ResponseHeaders: map[string]string{},
+ ResponseBody: "test",
+ ResponseBodyBytes: []byte("test"),
+ HasCustomResponseBody: true,
+ }
+
+ oc.HTTPClient = http.DefaultClient
+ br := bytes.NewBuffer([]byte("test"))
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", es.URL, br)
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(oc, pc, nil, nil, nil)))
+
+ //req := model.NewRequest("TestProxyRequest", r.Method, r.URL, http.Header{"testHeaderName": []string{"testHeaderValue"}}, time.Duration(30)*time.Second, r, tu.NewTestWebClient())
+ DoProxy(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), "test")
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestProxyRequestBadGateway(t *testing.T) {
+
+ const badUpstream = "http://127.0.0.1:64389"
+
+ // assume nothing listens on badUpstream, so this should force the proxy to generate a 502 Bad Gateway
+ err := config.Load("trickster", "test", []string{"-origin-url", badUpstream, "-origin-type", "test", "-log-level", "debug"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ oc := config.Origins["default"]
+ pc := &config.PathConfig{
+ Path: "/",
+ RequestHeaders: map[string]string{},
+ ResponseHeaders: map[string]string{},
+ }
+
+ oc.HTTPClient = http.DefaultClient
+ br := bytes.NewBuffer([]byte("test"))
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", badUpstream, br)
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(oc, pc, nil, nil, nil)))
+
+ //req := model.NewRequest("TestProxyRequest", r.Method, r.URL, make(http.Header), time.Duration(30)*time.Second, r, tu.NewTestWebClient())
+ DoProxy(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusBadGateway)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestClockOffsetWarning(t *testing.T) {
+
+ handler := func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Add(headers.NameDate, time.Now().Add(-1*time.Hour).Format(http.TimeFormat))
+ w.WriteHeader(200)
+ }
+ s := httptest.NewServer(http.HandlerFunc(handler))
+
+ err := config.Load("trickster", "test", []string{"-origin-url", s.URL, "-origin-type", "test", "-log-level", "debug"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ oc := config.Origins["default"]
+ pc := &config.PathConfig{
+ Path: "/",
+ RequestHeaders: map[string]string{},
+ ResponseHeaders: map[string]string{},
+ }
+
+ oc.HTTPClient = http.DefaultClient
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", s.URL, nil)
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(oc, pc, nil, nil, nil)))
+
+ if log.HasWarnedOnce("clockoffset.default") {
+ t.Errorf("expected %t got %t", false, true)
+ }
+
+ //req := model.NewRequest("TestProxyRequest", http.MethodGet, r.URL, make(http.Header), time.Duration(30)*time.Second, r, tu.NewTestWebClient())
+ DoProxy(w, r)
+ resp := w.Result()
+
+ if !log.HasWarnedOnce("clockoffset.default") {
+ t.Errorf("expected %t got %t", true, false)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+
+}
+
+func TestDoProxyWithPCF(t *testing.T) {
+
+ es := tu.NewTestServer(http.StatusOK, "test", nil)
+ defer es.Close()
+
+ err := config.Load("trickster", "test", []string{"-origin-url", es.URL, "-origin-type", "test", "-log-level", "debug"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ oc := config.Origins["default"]
+ pc := &config.PathConfig{
+ Path: "/",
+ RequestHeaders: map[string]string{},
+ ResponseHeaders: map[string]string{},
+ ResponseBody: "test",
+ ResponseBodyBytes: []byte("test"),
+ HasCustomResponseBody: true,
+ CollapsedForwardingName: "progressive",
+ CollapsedForwardingType: config.CFTypeProgressive,
+ }
+
+ oc.HTTPClient = http.DefaultClient
+ br := bytes.NewBuffer([]byte("test"))
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", es.URL, br)
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(oc, pc, nil, nil, nil)))
+
+ // get URL
+
+ //req := model.NewRequest("TestProxyRequest", r.Method, r.URL, http.Header{"testHeaderName": []string{"testHeaderValue"}}, time.Duration(30)*time.Second, r, tu.NewTestWebClient())
+ DoProxy(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), "test")
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestProxyRequestWithPCFMultipleClients(t *testing.T) {
+
+ es := tu.NewTestServer(http.StatusOK, "test", nil)
+ defer es.Close()
+
+ err := config.Load("trickster", "test", []string{"-origin-url", es.URL, "-origin-type", "test", "-log-level", "debug"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ oc := config.Origins["default"]
+ pc := &config.PathConfig{
+ Path: "/",
+ RequestHeaders: map[string]string{},
+ ResponseHeaders: map[string]string{},
+ ResponseBody: "test",
+ ResponseBodyBytes: []byte("test"),
+ HasCustomResponseBody: true,
+ CollapsedForwardingName: "progressive",
+ CollapsedForwardingType: config.CFTypeProgressive,
+ }
+
+ oc.HTTPClient = http.DefaultClient
+ br := bytes.NewBuffer([]byte("test"))
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", es.URL, br)
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(oc, pc, nil, nil, nil)))
+
+ // get URL
+
+ //req := model.NewRequest("TestProxyRequest", r.Method, r.URL, http.Header{"testHeaderName": []string{"testHeaderValue"}}, time.Duration(30)*time.Second, r, tu.NewTestWebClient())
+ DoProxy(w, r)
+ resp := w.Result()
+
+ err = testStatusCodeMatch(resp.StatusCode, http.StatusOK)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testStringMatch(string(bodyBytes), "test")
+ if err != nil {
+ t.Error(err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, map[string]string{"engine": "HTTPProxy"})
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestPrepareFetchReaderErr(t *testing.T) {
+
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://example.com/", "-origin-type", "test", "-log-level", "debug"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ oc := config.Origins["default"]
+ oc.HTTPClient = http.DefaultClient
+
+ r := httptest.NewRequest("GET", "http://example.com/", nil)
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(oc, nil, nil, nil, nil)))
+ r.Method = "\t"
+ _, _, i := PrepareFetchReader(r)
+ if i != 0 {
+ t.Errorf("expected 0 got %d", i)
+ }
+}
diff --git a/internal/proxy/engines/key.go b/internal/proxy/engines/key.go
new file mode 100644
index 000000000..e3673692a
--- /dev/null
+++ b/internal/proxy/engines/key.go
@@ -0,0 +1,165 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/url"
+ "sort"
+ "strconv"
+ "strings"
+
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/util/md5"
+)
+
+var methodsWithBody = map[string]bool{http.MethodPut: true, http.MethodPost: true, http.MethodPatch: true}
+
+// DeriveCacheKey calculates a query-specific keyname based on the prometheus query in the user request
+func (pr *proxyRequest) DeriveCacheKey(templateURL *url.URL, extra string) string {
+
+ rsc := request.GetResources(pr.Request)
+ pc := rsc.PathConfig
+
+ if pc == nil {
+ return md5.Checksum(pr.URL.Path + extra)
+ }
+
+ var params url.Values
+ r := pr.Request
+
+ if pr.upstreamRequest != nil {
+ r = pr.upstreamRequest
+ }
+
+ if templateURL != nil {
+ params = templateURL.Query()
+ } else if r.URL != nil {
+ params = r.URL.Query()
+ } else {
+ params = pr.URL.Query()
+ }
+
+ if pc.KeyHasher != nil && len(pc.KeyHasher) == 1 {
+ return pc.KeyHasher[0](pr.URL.Path, params, pr.Header, pr.Body, extra)
+ }
+
+ vals := make([]string, 0, (len(pc.CacheKeyParams) + len(pc.CacheKeyHeaders) + len(pc.CacheKeyFormFields)*2))
+
+ if v := pr.Header.Get(headers.NameAuthorization); v != "" {
+ vals = append(vals, fmt.Sprintf("%s.%s.", headers.NameAuthorization, v))
+ }
+
+ // Append the http method to the slice for creating the derived cache key
+ vals = append(vals, fmt.Sprintf("%s.%s.", "method", pr.Method))
+
+ if len(pc.CacheKeyParams) == 1 && pc.CacheKeyParams[0] == "*" {
+ for p := range params {
+ vals = append(vals, fmt.Sprintf("%s.%s.", p, params.Get(p)))
+ }
+ } else {
+ for _, p := range pc.CacheKeyParams {
+ if v := params.Get(p); v != "" {
+ vals = append(vals, fmt.Sprintf("%s.%s.", p, v))
+ }
+ }
+ }
+
+ for _, p := range pc.CacheKeyHeaders {
+ if v := pr.Header.Get(p); v != "" {
+ vals = append(vals, fmt.Sprintf("%s.%s.", p, v))
+ }
+ }
+
+ if _, ok := methodsWithBody[pr.Method]; ok && pc.CacheKeyFormFields != nil && len(pc.CacheKeyFormFields) > 0 {
+ ct := pr.Header.Get(headers.NameContentType)
+ if ct == headers.ValueXFormURLEncoded || strings.HasPrefix(ct, headers.ValueMultipartFormData) || ct == headers.ValueApplicationJSON {
+ b, _ := ioutil.ReadAll(pr.Body)
+ pr.Body = ioutil.NopCloser(bytes.NewReader(b))
+ if ct == headers.ValueXFormURLEncoded {
+ pr.ParseForm()
+ } else if strings.HasPrefix(ct, headers.ValueMultipartFormData) {
+ pr.ParseMultipartForm(1024 * 1024)
+ } else if ct == headers.ValueApplicationJSON {
+ var document map[string]interface{}
+ err := json.Unmarshal(b, &document)
+ if err == nil {
+ for _, f := range pc.CacheKeyFormFields {
+ v, err := deepSearch(document, f)
+ if err == nil {
+ if pr.Form == nil {
+ pr.Form = url.Values{}
+ }
+ pr.Form.Set(f, v)
+ }
+ }
+ }
+ }
+ pr.Body = ioutil.NopCloser(bytes.NewReader(b))
+ }
+
+ for _, f := range pc.CacheKeyFormFields {
+ if _, ok := pr.Form[f]; ok {
+ if v := pr.FormValue(f); v != "" {
+ vals = append(vals, fmt.Sprintf("%s.%s.", f, v))
+ }
+ }
+ }
+ }
+
+ sort.Strings(vals)
+ return md5.Checksum(pr.URL.Path + "." + strings.Join(vals, "") + extra)
+}
+
+func deepSearch(document map[string]interface{}, key string) (string, error) {
+
+ if key == "" {
+ return "", fmt.Errorf("invalid key name: %s", key)
+ }
+ parts := strings.Split(key, "/")
+ m := document
+ l := len(parts) - 1
+ for i, p := range parts {
+ v, ok := m[p]
+ if !ok {
+ return "", fmt.Errorf("could not find key: %s", key)
+ }
+ if l != i {
+ m, ok = v.(map[string]interface{})
+ if !ok {
+ return "", fmt.Errorf("could not find key: %s", key)
+ }
+ continue
+ }
+
+ if s, ok := v.(string); ok {
+ return s, nil
+ }
+
+ if i, ok := v.(float64); ok {
+ return strconv.FormatFloat(i, 'f', 4, 64), nil
+ }
+
+ if b, ok := v.(bool); ok {
+ return fmt.Sprintf("%t", b), nil
+ }
+
+ }
+ return "", fmt.Errorf("could not find key: %s", key)
+}
diff --git a/internal/proxy/engines/key_test.go b/internal/proxy/engines/key_test.go
new file mode 100644
index 000000000..913ba6504
--- /dev/null
+++ b/internal/proxy/engines/key_test.go
@@ -0,0 +1,244 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
+ "strconv"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/config"
+ ct "github.com/Comcast/trickster/internal/proxy/context"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+)
+
+const testMultipartBoundary = `; boundary=------------------------d0509edbe55938c0`
+const testMultipartBody = `--------------------------d0509edbe55938c0
+Content-Disposition: form-data; name="field1"
+
+value1
+--------------------------d0509edbe55938c0
+Content-Disposition: form-data; name="field2"
+
+value2
+--------------------------d0509edbe55938c0--
+`
+
+const testJSONDocument = `
+{
+ "requestType": "query",
+ "query": {
+ "table": "movies",
+ "fields": "eidr,title",
+ "filter": "year=1979",
+ "options": {
+ "batchSize": 20,
+ "someArray": [ "test" ],
+ "booleanHere": true
+ }
+ },
+ "field1": "value1"
+}
+`
+
+func TestDeepSearch(t *testing.T) {
+
+ var document map[string]interface{}
+ json.Unmarshal([]byte(testJSONDocument), &document)
+
+ val, err := deepSearch(document, "query/table")
+ if err != nil {
+ t.Error(err)
+ }
+
+ if val != "movies" {
+ t.Errorf("expected %s got %s", "movies", val)
+ }
+
+ _, err = deepSearch(document, "")
+ if err == nil {
+ t.Errorf("expected error: %s", "could not find key")
+ }
+
+ _, err = deepSearch(document, "missingKey")
+ if err == nil {
+ t.Errorf("expected error: %s", "could not find key")
+ }
+
+ _, err = deepSearch(document, "query/filter/nottamap")
+ if err == nil {
+ t.Errorf("expected error: %s", "could not find key")
+ }
+
+ _, err = deepSearch(document, "query/options/batchSize")
+ if err != nil {
+ t.Error(err)
+ }
+
+ _, err = deepSearch(document, "query/options/booleanHere")
+ if err != nil {
+ t.Error(err)
+ }
+
+ _, err = deepSearch(document, "query/options/someArray")
+ if err == nil {
+ t.Errorf("expected error: %s", "could not find key")
+ }
+}
+
+func TestDeriveCacheKey(t *testing.T) {
+
+ rpath := &config.PathConfig{
+ Path: "/",
+ CacheKeyParams: []string{"query", "step", "time"},
+ CacheKeyHeaders: []string{},
+ CacheKeyFormFields: []string{"field1"},
+ }
+
+ cfg := &config.OriginConfig{
+ Paths: map[string]*config.PathConfig{
+ "root": rpath,
+ },
+ }
+
+ tr := httptest.NewRequest("GET", "http://127.0.0.1/?query=12345&start=0&end=0&step=300&time=0", nil)
+ tr = tr.WithContext(ct.WithResources(context.Background(), request.NewResources(cfg, cfg.Paths["root"], nil, nil, nil)))
+
+ pr := newProxyRequest(tr, nil)
+ key := pr.DeriveCacheKey(nil, "extra")
+
+ if key != "52dc11456c84506d3444e53ee4c99777" {
+ t.Errorf("expected %s got %s", "52dc11456c84506d3444e53ee4c99777", key)
+ }
+
+ cfg.Paths["root"].CacheKeyParams = []string{"*"}
+
+ pr = newProxyRequest(tr, nil)
+ key = pr.DeriveCacheKey(pr.URL, "extra")
+ if key != "407aba34f02c87f6898a6d80b01f38a4" {
+ t.Errorf("expected %s got %s", "407aba34f02c87f6898a6d80b01f38a4", key)
+ }
+
+ const expected = "cb84ad010abb4d0f864470540a46f137"
+
+ tr = httptest.NewRequest(http.MethodPost, "http://127.0.0.1/", bytes.NewReader([]byte("field1=value1")))
+ tr = tr.WithContext(ct.WithResources(context.Background(), request.NewResources(cfg, cfg.Paths["root"], nil, nil, nil)))
+ tr.Header.Set(headers.NameContentType, headers.ValueXFormURLEncoded)
+ pr = newProxyRequest(tr, nil)
+ key = pr.DeriveCacheKey(nil, "extra")
+ if key != expected {
+ t.Errorf("expected %s got %s", expected, key)
+ }
+
+ tr = httptest.NewRequest(http.MethodPut, "http://127.0.0.1/", bytes.NewReader([]byte(testMultipartBody)))
+ tr = tr.WithContext(ct.WithResources(context.Background(), request.NewResources(cfg, cfg.Paths["root"], nil, nil, nil)))
+ tr.Header.Set(headers.NameContentType, headers.ValueMultipartFormData+testMultipartBoundary)
+ tr.Header.Set(headers.NameContentLength, strconv.Itoa(len(testMultipartBody)))
+ pr = newProxyRequest(tr, nil)
+ key = pr.DeriveCacheKey(nil, "extra")
+ if key != "4766201eee9ef1916f57309deae22f90" {
+ t.Errorf("expected %s got %s", "4766201eee9ef1916f57309deae22f90", key)
+ }
+
+ tr = httptest.NewRequest(http.MethodPost, "http://127.0.0.1/", bytes.NewReader([]byte(testJSONDocument)))
+ tr = tr.WithContext(ct.WithResources(context.Background(), request.NewResources(cfg, cfg.Paths["root"], nil, nil, nil)))
+ tr.Header.Set(headers.NameContentType, headers.ValueApplicationJSON)
+ tr.Header.Set(headers.NameContentLength, strconv.Itoa(len(testJSONDocument)))
+ pr = newProxyRequest(tr, nil)
+ pr.upstreamRequest.URL = nil
+ key = pr.DeriveCacheKey(nil, "extra")
+ if key != expected {
+ t.Errorf("expected %s got %s", expected, key)
+ }
+
+ // Test Custom KeyHasher Integration
+ rpath.KeyHasher = []config.KeyHasherFunc{exampleKeyHasher}
+ key = pr.DeriveCacheKey(nil, "extra")
+ if key != "test-key" {
+ t.Errorf("expected %s got %s", "test-key", key)
+ }
+
+}
+
+func exampleKeyHasher(path string, params url.Values, headers http.Header, body io.ReadCloser, extra string) string {
+ return "test-key"
+}
+
+func TestDeriveCacheKeyAuthHeader(t *testing.T) {
+
+ client := &TestClient{
+ config: &config.OriginConfig{
+ Paths: map[string]*config.PathConfig{
+ "root": {
+ Path: "/",
+ CacheKeyParams: []string{"query", "step", "time"},
+ CacheKeyHeaders: []string{"X-Test-Header"},
+ },
+ },
+ },
+ }
+
+ tr := httptest.NewRequest("GET", "http://127.0.0.1/?query=12345&start=0&end=0&step=300&time=0", nil)
+ tr = tr.WithContext(ct.WithResources(context.Background(),
+ request.NewResources(client.Configuration(), client.Configuration().Paths["root"], nil, nil, nil)))
+
+ tr.Header.Add("Authorization", "test")
+ tr.Header.Add("X-Test-Header", "test2")
+
+ pr := newProxyRequest(tr, nil)
+
+ //r := &model.Request{URL: u, TimeRangeQuery: ×eries.TimeRangeQuery{Step: 300000}, ClientRequest: tr}
+ //r.Headers = tr.Header
+
+ key := pr.DeriveCacheKey(nil, "extra")
+
+ if key != "60257fa6b18d6072b90a294269a8e6e1" {
+ t.Errorf("expected %s got %s", "60257fa6b18d6072b90a294269a8e6e1", key)
+ }
+
+}
+
+func TestDeriveCacheKeyNoPathConfig(t *testing.T) {
+
+ client := &TestClient{
+ config: &config.OriginConfig{
+ Paths: map[string]*config.PathConfig{
+ "root": {
+ Path: "/",
+ CacheKeyParams: []string{"query", "step", "time"},
+ CacheKeyHeaders: []string{},
+ },
+ },
+ },
+ }
+
+ tr := httptest.NewRequest("GET", "http://127.0.0.1/?query=12345&start=0&end=0&step=300&time=0", nil)
+ tr = tr.WithContext(ct.WithResources(context.Background(),
+ request.NewResources(client.Configuration(), nil, nil, nil, nil)))
+
+ pr := newProxyRequest(tr, nil)
+ key := pr.DeriveCacheKey(nil, "extra")
+
+ if key != "f53b04ce5c434a7357804ae15a64ee6c" {
+ t.Errorf("expected %s got %s", "f53b04ce5c434a7357804ae15a64ee6c", key)
+ }
+
+}
diff --git a/internal/proxy/engines/objectproxycache.go b/internal/proxy/engines/objectproxycache.go
new file mode 100644
index 000000000..7c651db28
--- /dev/null
+++ b/internal/proxy/engines/objectproxycache.go
@@ -0,0 +1,339 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "errors"
+ "io"
+ "io/ioutil"
+ "net/http"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/util/log"
+ "github.com/Comcast/trickster/pkg/locks"
+)
+
+func handleCacheKeyHit(pr *proxyRequest) error {
+
+ d := pr.cacheDocument
+
+ if d != nil && d.StoredRangeParts != nil && len(d.StoredRangeParts) > 0 {
+ d.LoadRangeParts()
+ }
+
+ ok, err := confirmTrueCacheHit(pr)
+ if ok {
+ return handleTrueCacheHit(pr)
+ }
+
+ // if not ok, then confirmTrueCacheHit already redirected the
+ // request to the correct handle; we just return its result here.
+ return err
+}
+
+func handleCachePartialHit(pr *proxyRequest) error {
+
+ // if we already have a revalidation in progress, then we've already confirmed it's not
+ // a true cache hit on the existing cached ranges. otherwise we need to verify first.
+ if pr.revalidation == RevalStatusNone {
+ if ok, err := confirmTrueCacheHit(pr); !ok {
+ // if not ok, then confirmTrueCacheHit has already redirected the
+ // request to the correct handle; we just return its result here.
+ return err
+ }
+ }
+
+ pr.prepareUpstreamRequests()
+
+ handleUpstreamTransactions(pr)
+
+ d := pr.cacheDocument
+ resp := pr.upstreamResponse
+ if pr.isPartialResponse {
+ b, _ := ioutil.ReadAll(pr.upstreamReader)
+ d2 := &HTTPDocument{}
+
+ d2.ParsePartialContentBody(resp, b)
+ d.LoadRangeParts()
+
+ d2.Ranges = d2.RangeParts.Ranges()
+
+ d.RangeParts.Merge(d2.RangeParts)
+ d.Ranges = d.RangeParts.Ranges()
+ d.StoredRangeParts = d.RangeParts.PackableMultipartByteRanges()
+ err := d.FulfillContentBody()
+
+ if err == nil {
+ pr.upstreamResponse.Body = ioutil.NopCloser(bytes.NewBuffer(d.Body))
+ pr.upstreamResponse.Header.Set(headers.NameContentType, d.ContentType)
+ pr.upstreamReader = pr.upstreamResponse.Body
+ } else {
+ h, b := d.RangeParts.ExtractResponseRange(pr.wantedRanges, d.ContentLength, d.ContentType, nil)
+
+ headers.Merge(pr.upstreamResponse.Header, h)
+ pr.upstreamReader = ioutil.NopCloser(bytes.NewBuffer(b))
+ }
+
+ } else {
+ if d != nil {
+ d.RangeParts = nil
+ d.Ranges = nil
+ d.StoredRangeParts = nil
+ d.StatusCode = resp.StatusCode
+ http.Header(d.Headers).Del(headers.NameContentRange)
+ }
+ }
+
+ pr.store()
+
+ return handleResponse(pr)
+
+}
+
+func confirmTrueCacheHit(pr *proxyRequest) (bool, error) {
+
+ pr.cachingPolicy.Merge(pr.cacheDocument.CachingPolicy)
+
+ if (!pr.checkCacheFreshness()) && (pr.cachingPolicy.CanRevalidate) {
+ return false, handleCacheRevalidation(pr)
+ }
+ if !pr.cachingPolicy.IsFresh {
+ pr.cacheStatus = status.LookupStatusKeyMiss
+ return false, handleCacheKeyMiss(pr)
+ }
+
+ return true, nil
+}
+
+func handleCacheRangeMiss(pr *proxyRequest) error {
+ // ultimately we can optimize range miss functionality compared to partial hit
+ // (e.g., if the object has expired, no need to revalidate on a range miss,
+ // but must dump old parts if the new range has a different etag or last-modified)
+ // for now we'll just treat it like partial hit, but it's still observed as a range miss
+ return handleCachePartialHit(pr)
+}
+
+func handleCacheRevalidation(pr *proxyRequest) error {
+
+ pr.revalidation = RevalStatusInProgress
+
+ // if it's a range miss, we don't need to remote revalidate.
+ // range miss means we have a cache key for this object, but
+ // not any of the byte ranges that the user has requested.
+ // since the needed range is 100% uncached, we can use
+ // the last-modified/etag of the new response to perform
+ // an internal revalidation of the pre-existing partial content.
+ if pr.cacheStatus == status.LookupStatusRangeMiss {
+ pr.revalidation = RevalStatusLocal
+ return handleCacheRangeMiss(pr)
+ }
+
+ // all other cache statuses that got us to this point means
+ // we have to perform a remote revalidation; quee it up
+ pr.prepareRevalidationRequest()
+
+ if pr.cacheStatus == status.LookupStatusPartialHit {
+ // this will handle all upstream calls including prepared reval
+ return handleCachePartialHit(pr)
+ }
+
+ // all remaining cache statuses indicate there are no other upstream
+ // requests than this revalidation. so lets make the call
+ handleUpstreamTransactions(pr)
+
+ return handleCacheRevalidationResponse(pr)
+
+}
+
+func handleCacheRevalidationResponse(pr *proxyRequest) error {
+
+ if pr.upstreamResponse.StatusCode == http.StatusNotModified {
+ pr.revalidation = RevalStatusOK
+ pr.cachingPolicy.IsFresh = true
+ pr.cachingPolicy.LocalDate = time.Now()
+ pr.cacheStatus = status.LookupStatusRevalidated
+ pr.upstreamResponse.StatusCode = pr.cacheDocument.StatusCode
+ pr.writeToCache = true
+ pr.store()
+ pr.upstreamReader = bytes.NewBuffer(pr.cacheDocument.Body)
+ return handleTrueCacheHit(pr)
+ }
+
+ pr.revalidation = RevalStatusFailed
+ pr.cacheStatus = status.LookupStatusKeyMiss
+ return handleAllWrites(pr)
+}
+
+func handleTrueCacheHit(pr *proxyRequest) error {
+
+ d := pr.cacheDocument
+ if d == nil {
+ return errors.New("nil cacheDocument")
+ }
+
+ if pr.cachingPolicy.IsNegativeCache {
+ pr.cacheStatus = status.LookupStatusNegativeCacheHit
+ }
+
+ pr.upstreamResponse = &http.Response{StatusCode: d.StatusCode, Request: pr.Request, Header: d.Headers}
+ if pr.wantsRanges {
+ h, b := d.RangeParts.ExtractResponseRange(pr.wantedRanges, d.ContentLength, d.ContentType, d.Body)
+ headers.Merge(pr.upstreamResponse.Header, h)
+ pr.upstreamReader = bytes.NewBuffer(b)
+ } else {
+ pr.upstreamReader = bytes.NewBuffer(d.Body)
+ }
+
+ return handleResponse(pr)
+
+}
+
+func handleCacheKeyMiss(pr *proxyRequest) error {
+
+ pr.prepareUpstreamRequests()
+
+ handleUpstreamTransactions(pr)
+
+ return handleAllWrites(pr)
+}
+
+func handleUpstreamTransactions(pr *proxyRequest) error {
+
+ pr.makeUpstreamRequests()
+ pr.reconstituteResponses()
+ pr.determineCacheability()
+
+ return nil
+}
+
+func handleAllWrites(pr *proxyRequest) error {
+ handleResponse(pr)
+ if pr.writeToCache {
+ if pr.cacheDocument == nil || !pr.cacheDocument.isLoaded {
+ d := DocumentFromHTTPResponse(pr.upstreamResponse, nil, pr.cachingPolicy)
+ pr.cacheDocument = d
+ if pr.isPartialResponse {
+ d.ParsePartialContentBody(pr.upstreamResponse, pr.cacheBuffer.Bytes())
+ } else {
+ d.Body = pr.cacheBuffer.Bytes()
+ }
+ }
+ pr.store()
+ }
+ return nil
+}
+
+func handleResponse(pr *proxyRequest) error {
+
+ pr.prepareResponse()
+ pr.writeResponseHeader()
+ pr.setBodyWriter() // what about partial hit? it does not set this
+ pr.writeResponseBody()
+ return nil
+}
+
+// Cache Status Response Handler Mappings
+var cacheResponseHandlers = map[status.LookupStatus]func(*proxyRequest) error{
+ status.LookupStatusHit: handleCacheKeyHit,
+ status.LookupStatusPartialHit: handleCachePartialHit,
+ status.LookupStatusKeyMiss: handleCacheKeyMiss,
+ status.LookupStatusRangeMiss: handleCacheRangeMiss,
+}
+
+func fetchViaObjectProxyCache(w io.Writer, r *http.Request) (*http.Response, status.LookupStatus) {
+
+ rsc := request.GetResources(r)
+ oc := rsc.OriginConfig
+ cc := rsc.CacheClient
+
+ pr := newProxyRequest(r, w)
+ pr.parseRequestRanges()
+
+ pr.cachingPolicy = GetRequestCachingPolicy(pr.Header)
+
+ pr.key = oc.CacheKeyPrefix + "." + pr.DeriveCacheKey(nil, "")
+ pcfResult, pcfExists := Reqs.Load(pr.key)
+ if (!pr.wantsRanges && pcfExists) || pr.cachingPolicy.NoCache {
+ if pr.cachingPolicy.NoCache {
+ locks.Acquire(pr.key)
+ cc.Remove(pr.key)
+ locks.Release(pr.key)
+ }
+ return nil, status.LookupStatusProxyOnly
+ }
+
+ if pcfExists {
+ pr.collapsedForwarder = pcfResult.(ProgressiveCollapseForwarder)
+ }
+
+ pr.cachingPolicy.ParseClientConditionals()
+
+ if !rsc.NoLock {
+ locks.Acquire(pr.key)
+ }
+
+ var err error
+ pr.cacheDocument, pr.cacheStatus, pr.neededRanges, err = QueryCache(cc, pr.key, pr.wantedRanges)
+ if err == nil || err == cache.ErrKNF {
+ if f, ok := cacheResponseHandlers[pr.cacheStatus]; ok {
+ f(pr)
+ } else {
+ log.Warn("unhandled cache lookup response", log.Pairs{"lookupStatus": pr.cacheStatus})
+ return nil, status.LookupStatusProxyOnly
+ }
+ } else {
+ log.Error("cache lookup error", log.Pairs{"detail": err.Error()})
+ pr.cacheDocument = nil
+ pr.cacheStatus = status.LookupStatusKeyMiss
+ handleCacheKeyMiss(pr)
+ }
+
+ if !rsc.NoLock {
+ locks.Release(pr.key)
+ }
+
+ // newProxyRequest sets pr.started to time.Now()
+ pr.elapsed = time.Since(pr.started)
+ el := float64(pr.elapsed.Milliseconds()) / 1000.0
+ recordOPCResult(r, pr.cacheStatus, pr.upstreamResponse.StatusCode, r.URL.Path, el, pr.upstreamResponse.Header)
+
+ return pr.upstreamResponse, pr.cacheStatus
+}
+
+// ObjectProxyCacheRequest provides a Basic HTTP Reverse Proxy/Cache
+func ObjectProxyCacheRequest(w http.ResponseWriter, r *http.Request) {
+ _, cacheStatus := fetchViaObjectProxyCache(w, r)
+ if cacheStatus == status.LookupStatusProxyOnly {
+ DoProxy(w, r)
+ }
+}
+
+// FetchViaObjectProxyCache Fetches an object from Cache or Origin (on miss), writes the object to the cache, and returns the object to the caller
+func FetchViaObjectProxyCache(r *http.Request) ([]byte, *http.Response, bool) {
+ w := bytes.NewBuffer(nil)
+ resp, cacheStatus := fetchViaObjectProxyCache(w, r)
+ if cacheStatus == status.LookupStatusProxyOnly {
+ resp = DoProxy(w, r)
+ }
+ return w.Bytes(), resp, cacheStatus == status.LookupStatusHit
+}
+
+func recordOPCResult(r *http.Request, cacheStatus status.LookupStatus, httpStatus int, path string, elapsed float64, header http.Header) {
+ recordResults(r, "ObjectProxyCache", cacheStatus, httpStatus, path, "", elapsed, nil, header)
+}
diff --git a/internal/proxy/engines/objectproxycache_test.go b/internal/proxy/engines/objectproxycache_test.go
new file mode 100644
index 000000000..288f5e709
--- /dev/null
+++ b/internal/proxy/engines/objectproxycache_test.go
@@ -0,0 +1,1202 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ tc "github.com/Comcast/trickster/internal/proxy/context"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+ "github.com/Comcast/trickster/pkg/rangesim"
+)
+
+func setupTestHarnessOPC(file, body string, code int, headers map[string]string) (*httptest.Server, *httptest.ResponseRecorder, *http.Request, *request.Resources, error) {
+ return setupTestHarnessOPCByType(file, "test", "/opc", body, code, headers)
+}
+
+func setupTestHarnessOPCRange(hdr map[string]string) (*httptest.Server, *httptest.ResponseRecorder, *http.Request, *request.Resources, error) {
+ s, rr, r, rsc, err := setupTestHarnessOPCByType("", "rangesim", "/opc", "", 0, hdr)
+ return s, rr, r, rsc, err
+}
+
+func setupTestHarnessOPCByType(
+ file, serverType, path, body string, code int, headers map[string]string,
+) (*httptest.Server, *httptest.ResponseRecorder, *http.Request, *request.Resources, error) {
+
+ client := &TestClient{}
+ ts, w, r, hc, err := tu.NewTestInstance(file, client.DefaultPathConfigs, code, body, headers, serverType, path, "debug")
+ if err != nil {
+ return nil, nil, nil, nil, fmt.Errorf("Could not load configuration: %s", err.Error())
+ }
+ r.RequestURI = ""
+ rsc := request.GetResources(r)
+ rsc.OriginClient = client
+ pc := rsc.PathConfig
+
+ if pc == nil {
+ return nil, nil, nil, nil, fmt.Errorf("could not find path %s", "/")
+ }
+
+ oc := rsc.OriginConfig
+ cc := rsc.CacheClient
+ oc.HTTPClient = hc
+
+ client.cache = cc
+ client.webClient = hc
+ client.config = oc
+
+ pc.CacheKeyParams = []string{"rangeKey", "instantKey"}
+
+ return ts, w, r, rsc, nil
+}
+
+func setupTestHarnessOPCWithPCF(file, body string, code int, headers map[string]string) (*httptest.Server, *httptest.ResponseRecorder, *http.Request, *request.Resources, error) {
+
+ client := &TestClient{}
+ ts, w, r, hc, err := tu.NewTestInstance(file, client.DefaultPathConfigs, code, body, headers, "prometheus", "/api/v1/query", "debug")
+ if err != nil {
+ return nil, nil, nil, nil, fmt.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ rsc := request.GetResources(r)
+ rsc.OriginClient = client
+ pc := rsc.PathConfig
+
+ if pc == nil {
+ return nil, nil, nil, nil, fmt.Errorf("could not find path %s", "/api/v1/query")
+ }
+
+ pc.CollapsedForwardingName = "progressive"
+ pc.CollapsedForwardingType = config.CFTypeProgressive
+
+ oc := rsc.OriginConfig
+ cc := rsc.CacheClient
+
+ oc.HTTPClient = hc
+ client.cache = cc
+ client.webClient = hc
+ client.config = oc
+
+ pc.CacheKeyParams = []string{"rangeKey", "instantKey"}
+
+ return ts, w, r, rsc, nil
+}
+
+func TestObjectProxyCacheRequest(t *testing.T) {
+
+ hdrs := map[string]string{"Cache-Control": "max-age=60"}
+ ts, _, r, rsc, err := setupTestHarnessOPC("", "test", http.StatusPartialContent, hdrs)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ r.Header.Add(headers.NameRange, "bytes=0-3")
+
+ oc := rsc.OriginConfig
+ oc.MaxTTLSecs = 15
+ oc.MaxTTL = time.Duration(oc.MaxTTLSecs) * time.Second
+
+ _, e := testFetchOPC(r, http.StatusPartialContent, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // get cache hit coverage too by repeating:
+ _, e = testFetchOPC(r, http.StatusPartialContent, "test", map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Remove Cache Hit from the Response Handler Map to test unknown handler error condition
+ delete(cacheResponseHandlers, status.LookupStatusHit)
+
+ _, e = testFetchOPC(r, http.StatusPartialContent, "test", map[string]string{"status": "proxy-only"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // add cache hit back
+ cacheResponseHandlers[status.LookupStatusHit] = handleCacheKeyHit
+
+}
+
+func TestObjectProxyCachePartialHit(t *testing.T) {
+ ts, _, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ // Cache miss on range
+ r.Header.Set(headers.NameRange, "bytes=0-10")
+ expectedBody, err := getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e := testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Partial Hit on an overlapping range
+ r.Header.Set(headers.NameRange, "bytes=5-15")
+
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Range Miss on an separate range
+ r.Header.Set(headers.NameRange, "bytes=60-70")
+
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "rmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Partial Hit on an multiple ranges
+ r.Header.Set(headers.NameRange, "bytes=10-20,50-55,60-65,69-75")
+ expectedBody, err = getExpectedRangeBody(r, "d5a5acd7eb4d3f622c62947a9904b89b")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Fulfill the cache with the remaining parts
+ r.Header.Del(headers.NameRange)
+ _, e = testFetchOPC(r, http.StatusOK, rangesim.Body, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Test Articulated Upstream
+ rsc.OriginConfig.DearticulateUpstreamRanges = true
+ r.Header.Set(headers.NameRange, "bytes=10-20,50-55,60-65,69-75")
+ r.URL.Path = "/new/test/path"
+ expectedBody, err = getExpectedRangeBody(r, "d5a5acd7eb4d3f622c62947a9904b89b")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestFullArticuation(t *testing.T) {
+
+ ts, _, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ // Test Articulated Upstream
+ rsc.OriginConfig.DearticulateUpstreamRanges = true
+ rsc.OriginConfig.RevalidationFactor = 2
+ r.Header.Set(headers.NameRange, "bytes=10-20,50-55,60-65,69-75")
+ r.URL.Path = "/new/test/path"
+ r.URL.RawQuery = "max-age=1"
+ expectedBody, err := getExpectedRangeBody(r, "d5a5acd7eb4d3f622c62947a9904b89b")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e := testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.RawQuery = "max-age=1&status=200"
+ r.URL.Path = "/new/test/path/2"
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.RawQuery = "max-age=1&ims=200"
+ r.URL.Path = "/new/test/path/3"
+ r.Header.Set(headers.NameRange, "bytes=10-20")
+ expectedBody, err = getExpectedRangeBody(r, "d5a5acd7eb4d3f622c62947a9904b89b")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1050)
+
+ r.Header.Set(headers.NameRange, "bytes=10-20, 25-30, 45-60")
+ expectedBody, err = getExpectedRangeBody(r, "a262725e1b8ae4967d369cff746e3924")
+ if err != nil {
+ t.Error(err)
+ }
+ r.URL.RawQuery = "max-age=1&ims=206"
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1050)
+
+ r.Header.Set(headers.NameRange, "bytes=9-20, 25-31, 42-65, 70-80")
+ expectedBody, err = getExpectedRangeBody(r, "34b73ea5c4c1ab5b9e34c9888119c58f")
+ if err != nil {
+ t.Error(err)
+ }
+ r.URL.RawQuery = "max-age=1&ims=206"
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1050)
+
+ r.Header.Set(headers.NameRange, "bytes=9-20, 90-95, 100-105")
+ expectedBody, err = getExpectedRangeBody(r, "01760208a2d6589fc9620627d561640d")
+ if err != nil {
+ t.Error(err)
+ }
+ r.URL.RawQuery = "max-age=1&ims=206"
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=9-20, 90-95, 100-105")
+ expectedBody, err = getExpectedRangeBody(r, "01760208a2d6589fc9620627d561640d")
+ if err != nil {
+ t.Error(err)
+ }
+ r.URL.Path = "/new/test/path/20"
+ r.URL.RawQuery = "max-age=1"
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1050)
+
+ r.Header.Set(headers.NameRange, "bytes=9-20, 25-32, 41-65")
+ expectedBody, err = getExpectedRangeBody(r, "722af19813169c99d8bda37a2f244f39")
+ if err != nil {
+ t.Error(err)
+ }
+ r.URL.RawQuery = "max-age=1&ims=206&non-ims=206"
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1050)
+
+ r.Header.Del(headers.NameRange)
+ r.URL.RawQuery = "max-age=1"
+ _, e = testFetchOPC(r, http.StatusOK, rangesim.Body, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=9-20, 21-22")
+ r.URL.Path = "/new/test/path/21"
+ expectedBody, err = getExpectedRangeBody(r, "368b9fbcef800068a48e70fa6e040289")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-1221")
+ r.URL.Path = "/new/test/path/22"
+ r.URL.RawQuery = ""
+ expectedBody, err = getExpectedRangeBody(r, "722af19813169c99d8bda37a2f244f39")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-1220,1221-1221")
+ expectedBody, err = getExpectedRangeBody(r, "0a6d16343fbe859a10cf1ac673e23dc9")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Del(headers.NameRange)
+ _, e = testFetchOPC(r, http.StatusOK, rangesim.Body, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+}
+
+func TestObjectProxyCachePartialHitNotFresh(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ pr := newProxyRequest(r, w)
+ oc := rsc.OriginConfig
+ cc := rsc.CacheClient
+ pr.cachingPolicy = GetRequestCachingPolicy(pr.Header)
+ pr.key = oc.Host + "." + pr.DeriveCacheKey(nil, "")
+ pr.cacheDocument, pr.cacheStatus, pr.neededRanges, _ = QueryCache(cc, pr.key, pr.wantedRanges)
+ handleCacheKeyMiss(pr)
+
+ pr.cachingPolicy.CanRevalidate = false
+ pr.cachingPolicy.IsFresh = false
+ pr.cachingPolicy.FreshnessLifetime = 0
+
+ pr.store()
+
+ handleCachePartialHit(pr)
+
+ if pr.isPartialResponse {
+ t.Errorf("Expected full response, got %t", pr.isPartialResponse)
+ }
+
+ if pr.cacheStatus != status.LookupStatusKeyMiss {
+ t.Errorf("Expected %s, got %s", status.LookupStatusKeyMiss, pr.cacheStatus)
+ }
+}
+
+func TestObjectProxyCachePartialHitFullResponse(t *testing.T) {
+
+ ts, w, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ pr := newProxyRequest(r, w)
+ oc := rsc.OriginConfig
+ cc := rsc.CacheClient
+ pr.cachingPolicy = GetRequestCachingPolicy(pr.Header)
+ pr.key = oc.Host + "." + pr.DeriveCacheKey(nil, "")
+ pr.cacheDocument, pr.cacheStatus, pr.neededRanges, _ = QueryCache(cc, pr.key, pr.wantedRanges)
+ handleCacheKeyMiss(pr)
+ handleCachePartialHit(pr)
+
+ if pr.isPartialResponse {
+ t.Errorf("Expected full response, got %t", pr.isPartialResponse)
+ }
+}
+
+func TestObjectProxyCacheRangeMiss(t *testing.T) {
+
+ ts, _, r, _, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ r.Header.Set(headers.NameRange, "bytes=0-10")
+ expectedBody, err := getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e := testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=15-20")
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "rmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheRevalidation(t *testing.T) {
+
+ ts, _, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ rsc.OriginConfig.RevalidationFactor = 2
+
+ r.Header.Set(headers.NameRange, "bytes=0-10")
+ if rsc.PathConfig == nil {
+ t.Error(errors.New("nil path config"))
+ }
+
+ if rsc.PathConfig.ResponseHeaders == nil {
+ rsc.PathConfig.ResponseHeaders = make(map[string]string)
+ }
+ rsc.PathConfig.ResponseHeaders[headers.NameCacheControl] = "max-age=1"
+
+ expectedBody, err := getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e := testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1010)
+
+ r.Header.Set(headers.NameRange, "bytes=0-10")
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "rhit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1010)
+ r.Header.Set(headers.NameRange, "bytes=0-15")
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // purge the cache
+ r.Header.Del(headers.NameRange)
+ r.Header.Set(headers.NameCacheControl, headers.ValueNoCache)
+
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusOK, expectedBody, map[string]string{"status": "proxy-only"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/10"
+
+ r.Header.Del(headers.NameCacheControl)
+
+ // now store it with an earlier last modified header
+ r.Header.Del(headers.NameCacheControl)
+ rsc.PathConfig.ResponseHeaders[headers.NameLastModified] = time.Unix(1577836799, 0).Format(time.RFC1123)
+ rsc.PathConfig.ResponseHeaders["-"+headers.NameCacheControl] = ""
+ rsc.PathConfig.ResponseHeaders[headers.NameExpires] = time.Now().Add(-1 * time.Minute).Format(time.RFC1123)
+
+ expectedBody, err = getExpectedRangeBody(r, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusOK, expectedBody, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // delete(rsc.PathConfig.ResponseHeaders, headers.NameLastModified)
+ // delete(rsc.PathConfig.ResponseHeaders, headers.NameExpires)
+
+ // expectedBody, err = getExpectedRangeBody(r, "")
+ // _, e = testFetchOPC(r, http.StatusOK, expectedBody, map[string]string{"status": "kmiss"})
+ // if e != nil {
+ // for _, err = range e {
+ // t.Error(err)
+ // }
+ // }
+}
+
+func TestObjectProxyCacheRequestWithPCF(t *testing.T) {
+
+ headers := map[string]string{"Cache-Control": "max-age=60"}
+ ts, _, r, rsc, err := setupTestHarnessOPCWithPCF("", "test", http.StatusOK, headers)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ oc := rsc.OriginConfig
+ oc.MaxTTLSecs = 15
+ oc.MaxTTL = time.Duration(oc.MaxTTLSecs) * time.Second
+
+ r.Header.Set("testHeaderName", "testHeaderValue")
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // get cache hit coverage too by repeating:
+ _, e = testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheTrueHitNoDocumentErr(t *testing.T) {
+
+ const expected = "nil cacheDocument"
+
+ pr := &proxyRequest{}
+ err := handleTrueCacheHit(pr)
+ if err.Error() != expected {
+ t.Errorf("expected %s got %s", expected, err.Error())
+ }
+}
+
+func TestObjectProxyCacheRequestClientNoCache(t *testing.T) {
+
+ ts, _, r, _, err := setupTestHarnessOPC("", "test", http.StatusOK, nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ r.Header.Set(headers.NameCacheControl, headers.ValueNoCache)
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "proxy-only"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestFetchViaObjectProxyCacheRequestClientNoCache(t *testing.T) {
+
+ ts, _, r, _, err := setupTestHarnessOPC("", "test", http.StatusOK, nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ r.Header.Set(headers.NameCacheControl, headers.ValueNoCache)
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "proxy-only"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ _, _, b := FetchViaObjectProxyCache(r)
+ if b {
+ t.Errorf("expected %t got %t", false, b)
+ }
+}
+
+func TestObjectProxyCacheRequestOriginNoCache(t *testing.T) {
+
+ headers := map[string]string{"Cache-Control": "no-cache"}
+ ts, _, r, _, err := setupTestHarnessOPC("", "test", http.StatusOK, headers)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheIMS(t *testing.T) {
+
+ hdrs := map[string]string{"Cache-Control": "max-age=1"}
+ ts, _, r, rsc, err := setupTestHarnessOPCRange(hdrs)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ rsc.OriginConfig.RevalidationFactor = 2
+
+ _, e := testFetchOPC(r, http.StatusOK, rangesim.Body, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameIfModifiedSince, "Wed, 01 Jan 2020 00:00:00 UTC")
+
+ _, e = testFetchOPC(r, http.StatusNotModified, "", map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(time.Millisecond * 1050)
+
+ r.URL.RawQuery = "status=200"
+
+ _, e = testFetchOPC(r, http.StatusNotModified, "", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ //t.Errorf("%s", "foo")
+
+}
+
+func TestObjectProxyCacheIUS(t *testing.T) {
+
+ // TODO: how does this test IUS???
+
+ headers := map[string]string{"Cache-Control": "max-age=60"}
+ ts, _, r, _, err := setupTestHarnessOPC("", "test", http.StatusOK, headers)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheINM(t *testing.T) {
+
+ rh := map[string]string{headers.NameCacheControl: "max-age=60", headers.NameETag: "test"}
+ ts, _, r, _, err := setupTestHarnessOPC("", "test", http.StatusOK, rh)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameIfNoneMatch, `"test"`)
+ _, e = testFetchOPC(r, http.StatusNotModified, "", map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameIfNoneMatch, `W/"test2"`)
+ _, e = testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheNoRevalidate(t *testing.T) {
+
+ headers := map[string]string{headers.NameCacheControl: headers.ValueMaxAge + "=1"}
+ ts, _, r, rsc, err := setupTestHarnessOPC("", "test", http.StatusOK, headers)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ p := rsc.PathConfig
+ p.ResponseHeaders = headers
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(1010 * time.Millisecond)
+
+ _, e = testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheCanRevalidate(t *testing.T) {
+
+ headers := map[string]string{
+ headers.NameCacheControl: headers.ValueMaxAge + "=1",
+ headers.NameETag: "test-etag",
+ }
+ ts, _, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ p := rsc.PathConfig
+ p.ResponseHeaders = headers
+ rsc.OriginConfig.RevalidationFactor = 2
+
+ _, e := testFetchOPC(r, http.StatusOK, rangesim.Body, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ time.Sleep(1010 * time.Millisecond)
+
+ _, e = testFetchOPC(r, http.StatusOK, rangesim.Body, map[string]string{"status": "rhit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheRevalidated(t *testing.T) {
+
+ const dt = "Sun, 16 Jun 2019 14:19:04 GMT"
+
+ hdr := map[string]string{
+ headers.NameCacheControl: headers.ValueMaxAge + "=2",
+ headers.NameLastModified: dt,
+ }
+ ts, _, r, rsc, err := setupTestHarnessOPC("", "test", http.StatusOK, hdr)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ rsc.PathConfig.ResponseHeaders = hdr
+
+ _, e := testFetchOPC(r, http.StatusOK, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameIfModifiedSince, dt)
+ _, e = testFetchOPC(r, http.StatusNotModified, "", map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestObjectProxyCacheRequestNegativeCache(t *testing.T) {
+
+ ts, _, r, rsc, err := setupTestHarnessOPC("", "test", http.StatusNotFound, nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ pc := config.NewPathConfig()
+ cfg := rsc.OriginConfig
+ cfg.Paths = map[string]*config.PathConfig{
+ "/": pc,
+ }
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(cfg, pc, rsc.CacheConfig, rsc.CacheClient, rsc.OriginClient)))
+
+ _, e := testFetchOPC(r, http.StatusNotFound, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // request again, should still cache miss, but this time, put 404's into the Negative Cache for 30s
+ cfg.NegativeCache[404] = time.Second * 30
+
+ _, e = testFetchOPC(r, http.StatusNotFound, "test", map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // request again, this time it should be a cache hit.
+ _, e = testFetchOPC(r, http.StatusNotFound, "test", map[string]string{"status": "nchit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+}
+
+func TestHandleCacheRevalidation(t *testing.T) {
+
+ ts, _, r, _, err := setupTestHarnessOPC("", "test", http.StatusNotFound, nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ pr := newProxyRequest(r, nil)
+ pr.cacheStatus = status.LookupStatusRangeMiss
+ pr.cachingPolicy = &CachingPolicy{}
+
+ err = handleCacheRevalidation(pr)
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func getExpectedRangeBody(r *http.Request, boundary string) (string, error) {
+
+ client := &http.Client{}
+ resp, err := client.Do(r)
+ if err != nil {
+ return "", err
+ }
+ b, _ := ioutil.ReadAll(resp.Body)
+ expectedBody := string(b)
+
+ if boundary != "" {
+ expectedBody = strings.Replace(expectedBody, "TestRangeServerBoundary", boundary, -1)
+ }
+
+ return expectedBody, nil
+}
+
+func TestRangesExhaustive(t *testing.T) {
+
+ ts, _, r, rsc, err := setupTestHarnessOPCRange(nil)
+ if err != nil {
+ t.Error(err)
+ }
+ defer ts.Close()
+
+ rsc.OriginConfig.RevalidationFactor = 2
+ rsc.OriginConfig.DearticulateUpstreamRanges = true
+
+ r.URL.Path = "/opc/test/1"
+ r.Header.Set(headers.NameRange, "bytes=0-6,25-32")
+ req := r.Clone(context.Background())
+ expectedBodyA, err := getExpectedRangeBody(req, "563a7014513fc6f0cbb4e8632dd107fc")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e := testFetchOPC(r, http.StatusPartialContent, expectedBodyA, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-10,20-28")
+ req = r.Clone(context.Background())
+ expectedBody, err := getExpectedRangeBody(req, "33f2477458123b02034bfbe20c52d949")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-6")
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "33f2477458123b02034bfbe20c52d949")
+ if err != nil {
+ t.Error(err)
+ }
+
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=5-7")
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "33f2477458123b02034bfbe20c52d949")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=29-29")
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "33f2477458123b02034bfbe20c52d949")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=9-22,28-60")
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "1fd80b6b357b4608027dd500ad3f3c21")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Del(headers.NameRange)
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "1fd80b6b357b4608027dd500ad3f3c21")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusOK, expectedBody, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-10,20-28")
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "33f2477458123b02034bfbe20c52d949")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-6")
+ req = r.Clone(context.Background())
+ expectedBody, err = getExpectedRangeBody(req, "33f2477458123b02034bfbe20c52d949")
+ if err != nil {
+ t.Error(err)
+ }
+
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody, map[string]string{"status": "hit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Test Range Revalidiations
+
+ rsc.PathConfig.ResponseHeaders = map[string]string{headers.NameCacheControl: "max-age=1"}
+
+ r.URL.Path = "/opc/test/2"
+ r.Header.Set(headers.NameRange, "bytes=0-6")
+ req = r.Clone(context.Background())
+ expectedBody1, err := getExpectedRangeBody(req, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody1, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/3"
+ r.Header.Set(headers.NameRange, "bytes=0-6, 8-10")
+ req = r.Clone(context.Background())
+ expectedBody2, err := getExpectedRangeBody(req, "1b4e59d25d723e317359c5e542d80f5c")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody2, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/4"
+ r.Header.Set(headers.NameRange, "bytes=0-6, 8-10")
+ req = r.Clone(context.Background())
+ expectedBody3, err := getExpectedRangeBody(req, "1b4e59d25d723e317359c5e542d80f5c")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody3, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/5"
+ r.Header.Set(headers.NameRange, "bytes=6-20")
+ req = r.Clone(context.Background())
+ expectedBody4, err := getExpectedRangeBody(req, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody4, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/6"
+ r.Header.Set(headers.NameRange, "bytes=6-20")
+ req = r.Clone(context.Background())
+ expectedBody5, err := getExpectedRangeBody(req, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody5, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/7"
+ r.Header.Set(headers.NameRange, "bytes=6-20")
+ req = r.Clone(context.Background())
+ expectedBody6, err := getExpectedRangeBody(req, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody6, map[string]string{"status": "kmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ // Now sleep to let them expire but not purge
+ time.Sleep(1050 * time.Millisecond)
+
+ // Now make more requests that require a revalidation first.
+
+ r.URL.Path = "/opc/test/2"
+ r.Header.Set(headers.NameRange, "bytes=0-6")
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody1, map[string]string{"status": "rhit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/3"
+ r.Header.Set(headers.NameRange, "bytes=0-6, 8-10")
+ expectedBody2 = strings.Replace(expectedBody2, "TestRangeServerBoundary", "1b4e59d25d723e317359c5e542d80f5c", -1)
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody2, map[string]string{"status": "rhit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/4"
+ r.Header.Set(headers.NameRange, "bytes=5-9")
+ req = r.Clone(context.Background())
+ expectedBody3, err = getExpectedRangeBody(req, "1b4e59d25d723e317359c5e542d80f5c")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody3, map[string]string{"status": "phit"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/5"
+ r.Header.Set(headers.NameRange, "bytes=0-5")
+ req = r.Clone(context.Background())
+ expectedBody4, err = getExpectedRangeBody(req, "")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody4, map[string]string{"status": "rmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/6"
+ r.Header.Set(headers.NameRange, "bytes=0-5,21-30")
+ req = r.Clone(context.Background())
+ expectedBody5, err = getExpectedRangeBody(req, "d51d39834c9650e17cc486c4a52cf572")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody5, map[string]string{"status": "rmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ r.URL.Path = "/opc/test/7"
+ r.Header.Set(headers.NameRange, "bytes=22-30,32-40")
+ req = r.Clone(context.Background())
+ expectedBody6, err = getExpectedRangeBody(req, "bab29463882afe6d6033e88dc74d2bdd")
+ if err != nil {
+ t.Error(err)
+ }
+ _, e = testFetchOPC(r, http.StatusPartialContent, expectedBody6, map[string]string{"status": "rmiss"})
+ for _, err = range e {
+ t.Error(err)
+ }
+
+ //rsc.OriginConfig.DearticulateUpstreamRanges = b
+
+ /*
+ √ curl -v --output - -H "Range: bytes=0-6,25-32" http://127.0.0.1:9091/rpc1/testing && \
+ √ curl -v --output - -H "Range: " http://127.0.0.1:9091/rpc1/testing
+ √ curl -v --output - -H "Range: bytes=0-6" http://127.0.0.1:9091/rpc1/testing && \
+ √ curl -v --output - -H "Range: bytes=5-7" http://127.0.0.1:9091/rpc1/testing && \
+ √ curl -v --output - -H "Range: bytes=29-29" http://127.0.0.1:9091/rpc1/testing && \
+ √ curl -v --output - -H "Range: bytes=9-22,28-60" http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - -H "Range: bytes=0-6" http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - -H "Range: bytes=0-6,10-20" http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - -H "Range: bytes=0-6, 10-19" http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - -H "Range: bytes=0-6,10-20" http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - http://127.0.0.1:9091/rpc1/testing
+ curl -v --output - -H "Range: bytes=0-6,7-1220" http://127.0.0.1:9091/rpc1/testing2 && \
+ curl -v --output - http://127.0.0.1:9091/rpc1/testing2 && \
+ curl -v --output - http://127.0.0.1:9091/rpc1/testing2
+ curl -v --output - -H "Range: bytes=0-6" http://127.0.0.1:9091/rpc1/testing3 && \
+ curl -v --output - -H "Range: bytes=5-20" http://127.0.0.1:9091/rpc1/testing3
+ curl -v --output - -H "Range: bytes=5-20" http://127.0.0.1:9091/rpc1/testing4 && \
+ curl -v --output - -H "Range: bytes=0-6" http://127.0.0.1:9091/rpc1/testing4
+
+ curl -v --output - -H "Range: bytes=0-6" http://127.0.0.1:9091/rpc1/testing && \
+ sleep 6 && curl -v --output - -H "Range: bytes=7-7" http://127.0.0.1:9091/rpc1/testing && \
+ curl -v --output - -H "Range: bytes=0-6" http://127.0.0.1:9091/rpc1/testing
+
+ */
+
+}
+
+func testFetchOPC(r *http.Request, sc int, body string, match map[string]string) (*httptest.ResponseRecorder, []error) {
+
+ e := make([]error, 0)
+
+ w := httptest.NewRecorder()
+
+ ObjectProxyCacheRequest(w, r)
+ resp := w.Result()
+
+ err := testStatusCodeMatch(resp.StatusCode, sc)
+ if err != nil {
+ e = append(e, err)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ e = append(e, err)
+ }
+
+ err = testStringMatch(string(bodyBytes), body)
+ if err != nil {
+ e = append(e, err)
+ }
+
+ err = testResultHeaderPartMatch(resp.Header, match)
+ if err != nil {
+ e = append(e, err)
+ }
+
+ if len(e) == 0 {
+ e = nil
+ }
+
+ return w, e
+
+}
diff --git a/internal/proxy/engines/progressive_collapse_forwarder.go b/internal/proxy/engines/progressive_collapse_forwarder.go
new file mode 100644
index 000000000..c47573a04
--- /dev/null
+++ b/internal/proxy/engines/progressive_collapse_forwarder.go
@@ -0,0 +1,195 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "errors"
+ "io"
+ "net/http"
+ "sync"
+ "sync/atomic"
+)
+
+// ErrReadIndexTooLarge is an error indicating the read index is too large
+var ErrReadIndexTooLarge = errors.New("Read index too large")
+
+// NEED TO DEAL WITH TIMEOUT
+
+// IndexReader implements a reader to read data at a specific index into slice b
+type IndexReader func(index uint64, b []byte) (int, error)
+
+// ProgressiveCollapseForwarder accepts data written through the io.Writer interface, caches it and
+// makes all the data written available to n readers. The readers can request data at index i,
+// to which the PCF may block or return the data immediately.
+type ProgressiveCollapseForwarder interface {
+ AddClient(io.Writer) error
+ Write([]byte) (int, error)
+ Close()
+ IndexRead(uint64, []byte) (int, error)
+ WaitServerComplete()
+ WaitAllComplete()
+ GetBody() ([]byte, error)
+ GetResp() *http.Response
+}
+
+type progressiveCollapseForwarder struct {
+ resp *http.Response
+ rIndex uint64
+ dataIndex uint64
+ data [][]byte
+ dataLen uint64
+ dataStore []byte
+ dataStoreLen uint64
+ readCond *sync.Cond
+ serverReadDone int32
+ clientWaitgroup *sync.WaitGroup
+ serverWaitCond *sync.Cond
+}
+
+// NewPCF returns a new instance of a ProgressiveCollapseForwarder
+func NewPCF(resp *http.Response, contentLength int64) ProgressiveCollapseForwarder {
+ // This contiguous block of memory is just an underlying byte store, references by the slices defined in refs
+ // Thread safety is provided through a read index, an atomic, which the writer must exceed and readers may not exceed
+ // This effectively limits the readers and writer to separate areas in memory.
+ dataStore := make([]byte, contentLength)
+ refs := make([][]byte, ((contentLength/HTTPBlockSize)*2)+1)
+
+ var wg sync.WaitGroup
+ sd := sync.NewCond(&sync.Mutex{})
+ rc := sync.NewCond(&sync.Mutex{})
+
+ pfc := &progressiveCollapseForwarder{
+ resp: resp,
+ rIndex: 0,
+ dataIndex: 0,
+ data: refs,
+ dataLen: uint64(len(refs)),
+ dataStore: dataStore,
+ dataStoreLen: uint64(contentLength),
+ readCond: rc,
+ serverReadDone: 0,
+ clientWaitgroup: &wg,
+ serverWaitCond: sd,
+ }
+
+ return pfc
+}
+
+// AddClient adds an io.Writer client to the ProgressiveCollapseForwarder
+// This client will read all the cached data and read from the live edge if caught up.
+func (pfc *progressiveCollapseForwarder) AddClient(w io.Writer) error {
+ pfc.clientWaitgroup.Add(1)
+ var readIndex uint64
+ var err error
+ remaining := 0
+ n := 0
+ buf := make([]byte, HTTPBlockSize)
+
+ for {
+ n, err = pfc.IndexRead(readIndex, buf)
+ if n > 0 {
+ // Handle the data returned by the read index > HTTPBlockSize
+ if n > HTTPBlockSize {
+ remaining = n
+ for {
+ if remaining > HTTPBlockSize {
+ w.Write(buf[0:HTTPBlockSize])
+ remaining -= HTTPBlockSize
+ } else {
+ w.Write(buf[0:remaining])
+ break
+ }
+ }
+ } else {
+ w.Write(buf[0:n])
+ }
+ readIndex++
+ }
+ if err != nil {
+ if err != io.EOF {
+ return err
+ }
+ break
+ }
+ }
+ pfc.clientWaitgroup.Done()
+ return io.EOF
+}
+
+// WaitServerComplete blocks until the object has been retrieved from the origin server
+// Need to get payload before can send to actual cache
+func (pfc *progressiveCollapseForwarder) WaitServerComplete() {
+ pfc.serverWaitCond.Wait()
+}
+
+// WaitAllComplete will wait till all clients have completed or timedout
+// Need to no abandon goroutines
+func (pfc *progressiveCollapseForwarder) WaitAllComplete() {
+ pfc.clientWaitgroup.Wait()
+}
+
+// GetBody returns the underlying body of the data written into a PCF
+func (pfc *progressiveCollapseForwarder) GetBody() ([]byte, error) {
+ if atomic.LoadInt32(&pfc.serverReadDone) == 0 {
+ return nil, errors.New("Server request not completed")
+ }
+ return pfc.dataStore[0:pfc.dataIndex], nil
+}
+
+// GetResp returns the response from the original request
+func (pfc *progressiveCollapseForwarder) GetResp() *http.Response {
+ return pfc.resp
+}
+
+// Write writes the data in b to the ProgressiveCollapseForwarders data store,
+// adds a reference to that data, and increments the read index.
+func (pfc *progressiveCollapseForwarder) Write(b []byte) (int, error) {
+ n := atomic.LoadUint64(&pfc.rIndex)
+ l := uint64(len(b))
+ if pfc.dataIndex+l > pfc.dataStoreLen || n > pfc.dataLen {
+ return 0, io.ErrShortWrite
+ }
+ pfc.data[n] = pfc.dataStore[pfc.dataIndex : pfc.dataIndex+l]
+ copy(pfc.data[n], b)
+ pfc.dataIndex += l
+ atomic.AddUint64(&pfc.rIndex, 1)
+ pfc.readCond.Broadcast()
+ return len(b), nil
+}
+
+// Close signals all things waiting on the server response body to complete.
+// This should be triggered by the client io.EOF
+func (pfc *progressiveCollapseForwarder) Close() {
+ atomic.AddInt32(&pfc.serverReadDone, 1)
+ pfc.serverWaitCond.Signal()
+ pfc.readCond.Broadcast()
+}
+
+// Read will return the given index data requested by the read is behind the PCF readindex, else blocks and waits for the data
+func (pfc *progressiveCollapseForwarder) IndexRead(index uint64, b []byte) (int, error) {
+ i := atomic.LoadUint64(&pfc.rIndex)
+ if index >= i {
+ // need to check completion and return io.EOF
+ if index > pfc.dataLen {
+ return 0, ErrReadIndexTooLarge
+ } else if atomic.LoadInt32(&pfc.serverReadDone) != 0 {
+ return 0, io.EOF
+ }
+ pfc.readCond.L.Lock()
+ pfc.readCond.Wait()
+ pfc.readCond.L.Unlock()
+ }
+ copy(b, pfc.data[index])
+ return len(pfc.data[index]), nil
+}
diff --git a/internal/proxy/engines/progressive_collapse_forwarder_test.go b/internal/proxy/engines/progressive_collapse_forwarder_test.go
new file mode 100644
index 000000000..509123cad
--- /dev/null
+++ b/internal/proxy/engines/progressive_collapse_forwarder_test.go
@@ -0,0 +1,263 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "io"
+ "net/http"
+ "reflect"
+ "strings"
+ "testing"
+)
+
+var testString = "Hey, I'm an http response body string."
+
+func TestPCFReadWriteSingle(t *testing.T) {
+ w := bytes.NewBuffer(make([]byte, 0, len(testString)))
+ r := strings.NewReader(testString)
+ l := len(testString)
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ var n int64
+ go func() {
+ n, _ = io.Copy(pcf, r)
+ pcf.Close()
+ }()
+ pcf.AddClient(w)
+
+ if n != int64(l) {
+ t.Errorf("PCF could not copy full length of reader")
+ }
+
+ if w.String() != testString {
+ t.Errorf("PCF result was not correct, expected: \"%s\" (Len: %d), got: \"%s\" (Len: %d)", testString, len(testString), w.String(), len(w.String()))
+ }
+}
+
+func TestPCFReadWriteMultiple(t *testing.T) {
+ w := bytes.NewBuffer(make([]byte, 0, len(testString)))
+ w1 := bytes.NewBuffer(make([]byte, 0, len(testString)))
+
+ r := strings.NewReader(testString)
+ l := len(testString)
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ var n int64
+ go func() {
+ n, _ = io.Copy(pcf, r)
+ pcf.Close()
+ }()
+ pcf.AddClient(w)
+ pcf.AddClient(w1)
+
+ if n != int64(l) {
+ t.Errorf("PCF could not copy full length of reader")
+ }
+
+ if w.String() != testString {
+ t.Errorf("PCF result was not correct, expected: \"%s\" (Len: %d), got: \"%s\" (Len: %d)", testString, len(testString), w.String(), len(w.String()))
+ }
+
+ if w1.String() != testString {
+ t.Errorf("PCF second client result was not correct, expected: \"%s\" (Len: %d), got: \"%s\" (Len: %d)", testString, len(testString), w1.String(), len(w1.String()))
+ }
+}
+
+func TestPCFReadWriteGetBody(t *testing.T) {
+ w := bytes.NewBuffer(make([]byte, 0, len(testString)))
+ r := strings.NewReader(testString)
+ l := len(testString)
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ var n int64
+
+ _, err := pcf.GetBody()
+ if err == nil {
+ t.Errorf("PCF expected an error on an unwritten body")
+ }
+
+ go func() {
+ n, _ = io.Copy(pcf, r)
+ pcf.Close()
+ }()
+ pcf.AddClient(w)
+
+ if n != int64(l) {
+ t.Errorf("PCF could not copy full length of reader")
+ }
+
+ if w.String() != testString {
+ t.Errorf("PCF result was not correct, expected: \"%s\" (Len: %d), got: \"%s\" (Len: %d)", testString, len(testString), w.String(), len(w.String()))
+ }
+
+ body, err := pcf.GetBody()
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(body) != testString {
+ t.Errorf("PCF result was not correct, expected: \"%s\" (Len: %d), got: \"%s\" (Len: %d)", testString, len(testString), string(body), len(body))
+ }
+}
+
+func TestPCFReadWriteClose(t *testing.T) {
+ w := bytes.NewBuffer(make([]byte, 0, len(testString)))
+ r := strings.NewReader(testString)
+ l := len(testString)
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ buf := make([]byte, 2)
+ n, _ := r.Read(buf)
+ pcf.Write(buf)
+ pcf.Close()
+ err := pcf.AddClient(w)
+
+ if err != io.EOF {
+ t.Errorf("PCF Close call did not return io.EOF")
+
+ }
+
+ if n != 2 {
+ t.Errorf("PCF Close read length incorrect, expected 2, got %d", n)
+ }
+}
+
+func TestPCFIndexReadTooLarge(t *testing.T) {
+ r := strings.NewReader(testString)
+ l := len(testString)
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ buf := make([]byte, 2)
+ r.Read(buf)
+ pcf.Write(buf)
+ pcf.Close()
+
+ _, err := pcf.IndexRead(12412, buf)
+
+ if err != ErrReadIndexTooLarge {
+ t.Errorf("PCF did not return ErrReadIndexTooLarge, got %e", err)
+ }
+}
+
+func TestPCFReadLarge(t *testing.T) {
+ r := bytes.NewBuffer(make([]byte, 64000))
+ w := bytes.NewBuffer(make([]byte, 64000))
+ l := r.Len()
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ var n int64
+ go func() {
+ n, _ = io.Copy(pcf, r)
+ pcf.Close()
+ }()
+ pcf.AddClient(w)
+
+ if n != int64(l) {
+ t.Errorf("PCF could not copy full length of reader")
+ }
+
+ if bytes.Equal(r.Bytes(), w.Bytes()) {
+ t.Errorf("PCF result was not correct, expected: \"%s\" (Len: %d), got: \"%s\" (Len: %d)", testString, len(testString), w.String(), len(w.String()))
+ }
+}
+
+func TestPCFResp(t *testing.T) {
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, 10)
+
+ if !reflect.DeepEqual(resp, pcf.GetResp()) {
+ t.Errorf("PCF GetResp failed to reproduce the original http response.")
+ }
+}
+
+func BenchmarkPCFWrite(b *testing.B) {
+ // 100MB object, simulated actual usecase sizes.
+ b.N = 3200
+
+ testBytes := make([]byte, 32*1024)
+ l := b.N * 32 * 1024
+ resp := &http.Response{}
+
+ pcf := NewPCF(resp, int64(l))
+ b.SetBytes(32 * 1024)
+
+ b.ResetTimer()
+ for n := 0; n < b.N; n++ {
+ pcf.Write(testBytes)
+ }
+}
+
+func BenchmarkPCFRead(b *testing.B) {
+ b.N = 3200
+
+ testBytes := make([]byte, 32*1024)
+ readBuf := make([]byte, 32*1024)
+
+ l := b.N * 32 * 1024
+ resp := &http.Response{}
+
+ var readIndex uint64
+ var err error
+
+ pcf := NewPCF(resp, int64(l))
+ b.SetBytes(32 * 1024)
+ for i := 0; i < b.N; i++ {
+ pcf.Write(testBytes)
+ }
+
+ b.ResetTimer()
+
+ for n := 0; n < b.N; n++ {
+ _, err = pcf.IndexRead(readIndex, readBuf)
+ readIndex++
+ if err != nil {
+ break
+ }
+ }
+}
+
+func BenchmarkPCFWriteRead(b *testing.B) {
+ b.N = 3200
+
+ testBytes := make([]byte, 32*1024)
+ readBuf := make([]byte, 32*1024)
+
+ l := b.N * 32 * 1024
+ resp := &http.Response{}
+
+ var readIndex uint64
+ var err error
+
+ pcf := NewPCF(resp, int64(l))
+ b.SetBytes(32 * 1024)
+
+ b.ResetTimer()
+
+ for n := 0; n < b.N; n++ {
+ pcf.Write(testBytes)
+ _, err = pcf.IndexRead(readIndex, readBuf)
+ readIndex++
+ if err != nil {
+ break
+ }
+ }
+}
diff --git a/internal/proxy/engines/proxy_request.go b/internal/proxy/engines/proxy_request.go
new file mode 100644
index 000000000..8c2ca7434
--- /dev/null
+++ b/internal/proxy/engines/proxy_request.go
@@ -0,0 +1,661 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "context"
+ "io"
+ "io/ioutil"
+ "net/http"
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+ "github.com/Comcast/trickster/internal/config"
+ tctx "github.com/Comcast/trickster/internal/proxy/context"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/util/log"
+)
+
+type proxyRequest struct {
+ *http.Request
+ responseWriter io.Writer
+ responseBody []byte
+
+ upstreamRequest *http.Request
+ upstreamResponse *http.Response
+ upstreamReader io.Reader
+
+ // for parallel requests
+ originRequests []*http.Request
+ originResponses []*http.Response
+ originReaders []io.ReadCloser
+
+ revalidationRequest *http.Request
+ revalidationResponse *http.Response
+ revalidationReader io.ReadCloser
+
+ cacheDocument *HTTPDocument
+ cacheBuffer *bytes.Buffer
+
+ key string
+ started time.Time
+ elapsed time.Duration
+ cacheStatus status.LookupStatus
+ writeToCache bool
+
+ wantsRanges bool
+ wantedRanges byterange.Ranges
+ neededRanges byterange.Ranges
+ rangeParts byterange.MultipartByteRanges
+
+ isPartialResponse bool
+ contentLength int64
+ revalidation RevalidationStatus
+ wasReconstituted bool
+ trueContentType string
+
+ collapsedForwarder ProgressiveCollapseForwarder
+ cachingPolicy *CachingPolicy
+}
+
+// newProxyRequest accepts the original inbound HTTP Request and Response
+// and returns a proxyRequest object
+func newProxyRequest(r *http.Request, w io.Writer) *proxyRequest {
+
+ pr := &proxyRequest{
+ Request: r,
+ upstreamRequest: r.Clone(context.Background()),
+ contentLength: -1,
+ responseWriter: w,
+ started: time.Now(),
+ }
+
+ rsc := request.GetResources(r)
+ pr.upstreamRequest = pr.upstreamRequest.WithContext(tctx.WithResources(pr.upstreamRequest.Context(), rsc))
+
+ return pr
+}
+
+func (pr *proxyRequest) Clone() *proxyRequest {
+ return &proxyRequest{
+ Request: pr.Request.Clone(context.Background()),
+ cacheDocument: pr.cacheDocument,
+ key: pr.key,
+ cacheStatus: pr.cacheStatus,
+ writeToCache: pr.writeToCache,
+ wantsRanges: pr.wantsRanges,
+ wantedRanges: pr.wantedRanges,
+ neededRanges: pr.neededRanges,
+ rangeParts: pr.rangeParts,
+ collapsedForwarder: pr.collapsedForwarder,
+ cachingPolicy: pr.cachingPolicy,
+ revalidation: pr.revalidation,
+ isPartialResponse: pr.isPartialResponse,
+ started: time.Now(),
+ }
+}
+
+// Fetch makes an HTTP request to the provided Origin URL, bypassing the Cache, and returns the
+// response and elapsed time to the caller.
+func (pr *proxyRequest) Fetch() ([]byte, *http.Response, time.Duration) {
+
+ rsc := request.GetResources(pr.Request)
+ oc := rsc.OriginConfig
+ pc := rsc.PathConfig
+
+ var handlerName string
+ if pc != nil {
+ handlerName = pc.HandlerName
+ }
+
+ start := time.Now()
+ reader, resp, _ := PrepareFetchReader(pr.Request)
+
+ var body []byte
+ var err error
+ if reader != nil {
+ body, err = ioutil.ReadAll(reader)
+ resp.Body.Close()
+ }
+ if err != nil {
+ log.Error("error reading body from http response", log.Pairs{"url": pr.URL.String(), "detail": err.Error()})
+ return []byte{}, resp, 0
+ }
+
+ elapsed := time.Since(start) // includes any time required to decompress the document for deserialization
+
+ if config.Logging.LogLevel == "debug" || config.Logging.LogLevel == "trace" {
+ go logUpstreamRequest(oc.Name, oc.OriginType, handlerName, pr.Method, pr.URL.String(), pr.UserAgent(), resp.StatusCode, len(body), elapsed.Seconds())
+ }
+
+ return body, resp, elapsed
+}
+
+func (pr *proxyRequest) prepareRevalidationRequest() {
+
+ pr.revalidation = RevalStatusInProgress
+ pr.revalidationRequest = request.SetResources(pr.upstreamRequest.Clone(context.Background()), request.GetResources(pr.Request))
+
+ if pr.cacheStatus == status.LookupStatusPartialHit {
+ var rh string
+ d := pr.cacheDocument
+ cl := d.ContentLength
+
+ rsc := request.GetResources(pr.Request)
+ // revalRanges are the ranges we have in cache that have expired, but the user needs
+ // so we revalidate these ranges in parallel with fetching of the uncached ranges
+
+ var wr byterange.Ranges
+
+ if pr.wantedRanges != nil && len(pr.wantedRanges) > 0 {
+ wr = pr.wantedRanges
+ } else {
+ wr = byterange.Ranges{{Start: 0, End: cl}}
+ }
+
+ revalRanges := wr.CalculateDelta(pr.neededRanges, cl)
+ l := len(revalRanges)
+ if (l > 1 && rsc.OriginConfig.DearticulateUpstreamRanges) && len(pr.cacheDocument.Ranges) == 1 {
+ rh = pr.cacheDocument.Ranges.String()
+ } else if l == 1 {
+ rh = revalRanges.String()
+ }
+
+ if rh != "" {
+ pr.revalidationRequest.Header.Set(headers.NameRange, rh)
+ } else {
+ pr.revalidationRequest.Header.Del(headers.NameRange)
+ }
+
+ }
+
+ if pr.cachingPolicy.ETag != "" {
+ pr.revalidationRequest.Header.Set(headers.NameIfNoneMatch, pr.cachingPolicy.ETag)
+ }
+ if !pr.cachingPolicy.LastModified.IsZero() {
+ pr.revalidationRequest.Header.Set(headers.NameIfModifiedSince, pr.cachingPolicy.LastModified.Format(time.RFC1123))
+ }
+
+}
+
+func (pr *proxyRequest) setRangeHeader(h http.Header) {
+ if pr.neededRanges != nil && len(pr.neededRanges) > 0 {
+ pr.cachingPolicy.IsFresh = false
+ h.Set(headers.NameRange, pr.neededRanges.String())
+ }
+}
+
+func (pr *proxyRequest) prepareUpstreamRequests() {
+
+ pr.setRangeHeader(pr.upstreamRequest.Header)
+
+ pr.stripConditionalHeaders()
+ rsc := request.GetResources(pr.Request)
+ if pr.originRequests == nil {
+ var l int
+ if pr.neededRanges == nil {
+ l = 1
+ } else {
+ l = len(pr.neededRanges)
+ }
+ pr.originRequests = make([]*http.Request, 0, l)
+ }
+
+ rsc.OriginConfig.DearticulateUpstreamRanges = true
+ // if we are articulating the origin range requests, break those out here
+ if pr.neededRanges != nil && len(pr.neededRanges) > 0 && rsc.OriginConfig.DearticulateUpstreamRanges {
+ for _, r := range pr.neededRanges {
+ req := request.SetResources(pr.upstreamRequest.Clone(context.Background()), rsc)
+ req.Header.Set(headers.NameRange, "bytes="+r.String())
+ pr.originRequests = append(pr.originRequests, req)
+ }
+ } else { // otherwise it will just be a list of one request.
+ pr.originRequests = []*http.Request{pr.upstreamRequest}
+ }
+}
+
+func (pr *proxyRequest) makeUpstreamRequests() error {
+
+ wg := sync.WaitGroup{}
+
+ if pr.revalidationRequest != nil {
+ wg.Add(1)
+ go func() {
+ pr.revalidationReader, pr.revalidationResponse, _ = PrepareFetchReader(pr.revalidationRequest)
+ wg.Done()
+ }()
+ }
+
+ if pr.originRequests != nil && len(pr.originRequests) > 0 {
+ pr.originResponses = make([]*http.Response, len(pr.originRequests))
+ pr.originReaders = make([]io.ReadCloser, len(pr.originRequests))
+ for i := range pr.originRequests {
+ wg.Add(1)
+ go func(j int) {
+ pr.originReaders[j], pr.originResponses[j], _ = PrepareFetchReader(pr.originRequests[j])
+ wg.Done()
+ }(i)
+ }
+ }
+
+ wg.Wait()
+
+ return nil
+}
+
+func (pr *proxyRequest) checkCacheFreshness() bool {
+ cp := pr.cachingPolicy
+ if pr.cachingPolicy == nil {
+ return false
+ }
+ cp.IsFresh = !cp.LocalDate.Add(time.Duration(cp.FreshnessLifetime) * time.Second).Before(time.Now())
+ return cp.IsFresh
+}
+
+func (pr *proxyRequest) parseRequestRanges() bool {
+ // handle byte range requests
+ var out byterange.Ranges
+ if _, ok := pr.Header[headers.NameRange]; ok {
+ out = byterange.ParseRangeHeader(pr.Header.Get(headers.NameRange))
+ }
+ pr.wantsRanges = out != nil && len(out) > 0
+ pr.wantedRanges = out
+
+ // if the client shouldn't support multipart ranges, force a full range
+ rsc := request.GetResources(pr.Request)
+ if rsc.OriginConfig.MultipartRangesDisabled && len(pr.wantedRanges) > 1 {
+ pr.upstreamRequest.Header.Del(headers.NameRange)
+ pr.wantsRanges = false
+ pr.wantedRanges = nil
+ }
+
+ return pr.wantsRanges
+}
+
+func (pr *proxyRequest) stripConditionalHeaders() {
+ // don't proxy these up, their scope is only between Trickster and client
+ if pr.cachingPolicy != nil && pr.cachingPolicy.IsClientConditional {
+ stripConditionalHeaders(pr.upstreamRequest.Header)
+ }
+}
+
+func (pr *proxyRequest) writeResponseHeader() {
+ headers.SetResultsHeader(pr.upstreamResponse.Header, "ObjectProxyCache", pr.cacheStatus.String(), "", nil)
+}
+
+func (pr *proxyRequest) setBodyWriter() {
+
+ PrepareResponseWriter(pr.responseWriter, pr.upstreamResponse.StatusCode, pr.upstreamResponse.Header)
+
+ if pr.writeToCache && pr.cacheBuffer == nil {
+ pr.cacheBuffer = &bytes.Buffer{}
+
+ if pr.cachingPolicy.IsClientFresh {
+ // don't write response body to the client on a 304 Not Modified
+ pr.responseWriter = pr.cacheBuffer
+ if pr.upstreamResponse.StatusCode == http.StatusNotModified {
+ pr.upstreamResponse.StatusCode = http.StatusOK
+ }
+ } else {
+ // we need to write to both the client over the wire, and the cache buffer
+ pr.responseWriter = io.MultiWriter(pr.responseWriter, pr.cacheBuffer)
+ }
+ } else if pr.upstreamResponse.StatusCode == http.StatusNotModified {
+ pr.responseWriter = nil
+ }
+}
+
+func (pr *proxyRequest) writeResponseBody() {
+ if pr.upstreamReader == nil || pr.responseWriter == nil {
+ return
+ }
+ io.Copy(pr.responseWriter, pr.upstreamReader)
+
+}
+
+func (pr *proxyRequest) determineCacheability() {
+
+ rsc := request.GetResources(pr.Request)
+ resp := pr.upstreamResponse
+
+ if resp != nil && resp.StatusCode >= 400 {
+ pr.writeToCache = pr.cachingPolicy.IsNegativeCache
+ resp.Header.Del(headers.NameCacheControl)
+ resp.Header.Del(headers.NameExpires)
+ resp.Header.Del(headers.NameLastModified)
+ resp.Header.Del(headers.NameETag)
+ resp.Header.Del(headers.NameContentLength)
+ return
+ }
+
+ if pr.revalidation == RevalStatusLocal {
+
+ tpc := pr.cachingPolicy.Clone()
+ tpc.IfModifiedSinceTime = pr.cacheDocument.CachingPolicy.LastModified
+ tpc.IfNoneMatchValue = pr.cacheDocument.CachingPolicy.ETag
+ tpc.IsClientConditional = true
+ tpc.ResolveClientConditionals(pr.cacheStatus)
+ if !tpc.IsClientFresh {
+ // this this case the range miss becomes a key miss since the old range failed revalidation
+ pr.cacheStatus = status.LookupStatusKeyMiss
+ pr.cacheDocument = nil
+ }
+ }
+
+ if rsc.AlternateCacheTTL > 0 {
+ pr.writeToCache = true
+ pr.cachingPolicy = &CachingPolicy{LocalDate: time.Now(), FreshnessLifetime: int(rsc.AlternateCacheTTL.Seconds())}
+ return
+ }
+
+ if pr.cachingPolicy.NoCache || (!pr.cachingPolicy.CanRevalidate && pr.cachingPolicy.FreshnessLifetime <= 0) {
+ pr.writeToCache = false
+ rsc.CacheClient.Remove(pr.key)
+ // is fresh, and we can cache, can revalidate and the freshness is greater than 0
+ } else if !pr.cachingPolicy.IsFresh {
+ pr.writeToCache = true
+ }
+}
+
+func (pr *proxyRequest) store() error {
+
+ if !pr.writeToCache || pr.cacheDocument == nil {
+ return nil
+ }
+
+ d := pr.cacheDocument
+
+ pr.writeToCache = false // in case store is called again before the object has changed
+
+ d.StoredRangeParts = d.RangeParts.PackableMultipartByteRanges()
+
+ if pr.trueContentType != "" {
+ pr.Header.Del(headers.NameContentType)
+ http.Header(d.Headers).Del(headers.NameContentType)
+ d.ContentType = pr.trueContentType
+ }
+
+ rsc := request.GetResources(pr.Request)
+ oc := rsc.OriginConfig
+
+ rf := oc.RevalidationFactor
+ if rsc.AlternateCacheTTL > 0 {
+ rf = 1
+ }
+
+ d.CachingPolicy = pr.cachingPolicy
+ err := WriteCache(rsc.CacheClient, pr.key, d, pr.cachingPolicy.TTL(rf, oc.MaxTTL), oc.CompressableTypes)
+ if err != nil {
+ return err
+ }
+ return nil
+}
+
+func (pr *proxyRequest) updateContentLength() {
+
+ resp := pr.upstreamResponse
+ if resp == nil || pr.responseBody == nil || pr.upstreamResponse.StatusCode > 299 {
+ return
+ }
+
+ resp.Header.Del(headers.NameContentLength)
+ pr.contentLength = int64(len(pr.responseBody))
+ resp.ContentLength = pr.contentLength
+
+ pr.upstreamReader = bytes.NewReader(pr.responseBody)
+}
+
+func (pr *proxyRequest) prepareResponse() {
+
+ pr.cachingPolicy.ResolveClientConditionals(pr.cacheStatus)
+
+ d := pr.cacheDocument
+ resp := pr.upstreamResponse
+
+ // if all of the client conditional headers were satisfied,
+ // return 304
+ if pr.cachingPolicy.IsClientFresh {
+ // 304 on an If-None-Match only applies to GET/HEAD requests
+ // this bit will convert an INM-based 304 to a 412 on non-GET/HEAD
+ if (pr.Method != http.MethodGet && pr.Method != http.MethodHead) &&
+ pr.cachingPolicy.HasIfNoneMatch && !pr.cachingPolicy.IfNoneMatchResult {
+ pr.upstreamResponse.StatusCode = http.StatusPreconditionFailed
+ } else {
+ resp.StatusCode = http.StatusNotModified
+ }
+ pr.responseBody = []byte{}
+ pr.updateContentLength()
+
+ return
+ }
+
+ if pr.wantsRanges && (resp.StatusCode == http.StatusOK || resp.StatusCode == http.StatusPartialContent) {
+
+ // since the user wants ranges, we have to extract them from what we have already
+ if !d.isLoaded &&
+ (pr.cacheStatus == status.LookupStatusPartialHit || pr.cacheStatus == status.LookupStatusKeyMiss || pr.cacheStatus == status.LookupStatusRangeMiss) {
+ var b []byte
+ if pr.upstreamReader != nil {
+ b, _ = ioutil.ReadAll(pr.upstreamReader)
+ }
+ d = DocumentFromHTTPResponse(pr.upstreamResponse, b, pr.cachingPolicy)
+ pr.cacheBuffer = bytes.NewBuffer(b)
+ if pr.writeToCache {
+ d.isLoaded = true
+ pr.cacheDocument = d
+ }
+ }
+
+ // we will need to stitch in a temporary content type header if it is a multipart response,
+ // but need the original content type and length if we are also writing to the cache
+ pr.trueContentType = resp.Header.Get(headers.NameContentType)
+ pr.contentLength = d.ContentLength
+
+ resp.StatusCode = http.StatusPartialContent
+
+ if d.Ranges != nil && len(d.Ranges) > 0 {
+ d.LoadRangeParts()
+ }
+ var h http.Header
+ pr.trueContentType = d.ContentType
+ h, pr.responseBody = d.RangeParts.ExtractResponseRange(pr.wantedRanges, d.ContentLength, d.ContentType, d.Body)
+ headers.Merge(resp.Header, h)
+ pr.upstreamReader = bytes.NewBuffer(pr.responseBody)
+ } else if !pr.wantsRanges {
+ if resp.StatusCode == http.StatusPartialContent {
+ resp.StatusCode = http.StatusOK
+ }
+ resp.Header.Del(headers.NameContentRange)
+ if pr.cacheStatus == status.LookupStatusHit || pr.cacheStatus == status.LookupStatusRevalidated || pr.cacheStatus == status.LookupStatusPartialHit {
+ pr.responseBody = d.Body
+ }
+ }
+
+ pr.updateContentLength()
+
+}
+
+// reconstitute will arrange and process multiple responses so that
+// we have just one response for the initial request
+func (pr *proxyRequest) reconstituteResponses() {
+
+ // rsc1 := request.GetResources(pr.Request)
+
+ hasRevalidationRequest := pr.revalidationRequest != nil
+
+ var wasRevalidated bool
+ if hasRevalidationRequest {
+ pr.upstreamRequest = pr.revalidationRequest
+ pr.upstreamResponse = pr.revalidationResponse
+ pr.upstreamReader = pr.upstreamResponse.Body
+ wasRevalidated = hasRevalidationRequest && pr.revalidationResponse.StatusCode == http.StatusNotModified
+ }
+
+ var originCount int
+ if pr.originRequests != nil {
+ originCount = len(pr.originRequests)
+ }
+
+ var requestCount int
+ if hasRevalidationRequest && !wasRevalidated {
+ requestCount = originCount + 1
+ } else {
+ requestCount = originCount
+ }
+
+ if requestCount == 0 && !hasRevalidationRequest {
+ return
+ }
+ // if we have a revalidation request, and its response is a 200 OK, or is the only upstream request
+ // we will set the primary source response to the revalidation response
+ if hasRevalidationRequest &&
+ (originCount == 0 || pr.revalidationResponse.StatusCode == http.StatusOK) {
+ requestCount = 1
+ } else if (!hasRevalidationRequest || wasRevalidated) && originCount == 1 {
+ // if we only have a single request, and it's a normal originRequest, set that to the response
+ // or if we had a revalidation request that was revalidated, and only one other origin request
+ pr.upstreamRequest = pr.originRequests[0]
+ pr.upstreamResponse = pr.originResponses[0]
+ pr.upstreamReader = pr.originResponses[0].Body
+ requestCount = 1
+ }
+
+ // if the revalidation request 304'd, we actually don't have to do anything else with it here.
+ hasRevalidationRequest = hasRevalidationRequest && !wasRevalidated
+
+ // first pass to handle any potential 200 OKs that should trump all other part-based responses
+ if requestCount > 1 {
+ for i := range pr.originRequests {
+ if pr.originResponses[i].StatusCode == http.StatusOK {
+ pr.upstreamRequest = pr.originRequests[i]
+ pr.upstreamResponse = pr.originResponses[i]
+ pr.upstreamReader = pr.originResponses[i].Body
+ //pr.cachingPolicy = &CachingPolicy{}
+ pr.upstreamResponse.Header.Del(headers.NameContentRange)
+ requestCount = 1
+ break
+ }
+ }
+ }
+
+ // if all requests were 206, we have to reconstitute to a single multipart body
+ pr.wasReconstituted = requestCount > 1
+
+ if pr.wasReconstituted {
+
+ // in this case, we should _not_ use the revalidation request as the base upstreamResponse,
+ // since it could have a 304 not modified as the response, instead of a 200 or 206, and this
+ // point assumes fresh
+
+ pr.upstreamReader = nil
+ pr.upstreamResponse = nil
+
+ appendLock := sync.Mutex{}
+ wg := sync.WaitGroup{}
+ parts := &HTTPDocument{}
+
+ if hasRevalidationRequest {
+ // if one of the parallel requests was a revalidation, it means the part we have in cache has expired.
+ // StatusCode will be: 1) 304 Not Modified (the entire cache is still fresh), 2) 206 Partial Content
+ // (cache is stale, returned range is the user-requested range that was stale cached, ready to serve
+ // fresh from the origin (we already handled the case of a 200 further up)
+ resp := pr.revalidationResponse
+
+ // if it's a 304 Not Modified, just don't do anything, since the cached document is good as-is, and
+ // the new responses below will returned to be merged with the existing cache. so we only check for 206 here.
+ if resp.StatusCode == http.StatusPartialContent {
+ wg.Add(1)
+ go func() {
+ // oh snap. so we have some partial content to merge in, but the original cache document
+ // is now invalid. lets go ahead and reset it.
+ b, _ := ioutil.ReadAll(resp.Body)
+ appendLock.Lock()
+ parts.ParsePartialContentBody(resp, b)
+ appendLock.Unlock()
+ wg.Done()
+ }()
+ }
+ }
+
+ for i := range pr.originRequests {
+ wg.Add(1)
+ go func(j int) {
+ r := pr.originRequests[j]
+ resp := pr.originResponses[j]
+
+ if pr.upstreamResponse == nil {
+ // only set the upstream response
+ appendLock.Lock()
+ if pr.upstreamResponse == nil {
+ pr.upstreamRequest = r
+ pr.upstreamResponse = resp
+ }
+ appendLock.Unlock()
+ }
+
+ if resp.StatusCode == http.StatusPartialContent {
+ b, _ := ioutil.ReadAll(resp.Body)
+ appendLock.Lock()
+ parts.ParsePartialContentBody(resp, b)
+ appendLock.Unlock()
+ }
+ wg.Done()
+ }(i)
+ }
+
+ // all the response bodies are loading in parallel. Wait until they are done.
+ wg.Wait()
+
+ resp := pr.upstreamResponse
+
+ parts.Ranges = parts.RangeParts.Ranges()
+
+ bodyFromParts := false
+ if len(parts.Ranges) > 0 {
+ resp.Header.Del(headers.NameContentRange)
+ pr.trueContentType = parts.ContentType
+ if bodyFromParts = len(parts.Ranges) > 1; !bodyFromParts {
+ err := parts.FulfillContentBody()
+ if bodyFromParts = err != nil; !bodyFromParts {
+ pr.upstreamReader = bytes.NewBuffer(parts.Body)
+ resp.StatusCode = http.StatusOK
+ pr.cacheBuffer = bytes.NewBuffer(parts.Body)
+ }
+ }
+ } else {
+ pr.upstreamReader = bytes.NewBuffer(parts.Body)
+ }
+
+ if bodyFromParts {
+ h, b := parts.RangeParts.Body(parts.ContentLength, parts.ContentType)
+ headers.Merge(resp.Header, h)
+ pr.upstreamReader = bytes.NewBuffer(b)
+ }
+ }
+
+ pr.isPartialResponse = pr.upstreamResponse.StatusCode == http.StatusPartialContent
+
+ // now we merge the caching policy of the new upstreams
+ if pr.upstreamResponse.StatusCode != http.StatusNotModified {
+ rsc := request.GetResources(pr.Request)
+ pr.cachingPolicy.Merge(GetResponseCachingPolicy(pr.upstreamResponse.StatusCode,
+ rsc.OriginConfig.NegativeCache, pr.upstreamResponse.Header))
+
+ }
+
+}
diff --git a/internal/proxy/engines/proxy_request_test.go b/internal/proxy/engines/proxy_request_test.go
new file mode 100644
index 000000000..23acf5a08
--- /dev/null
+++ b/internal/proxy/engines/proxy_request_test.go
@@ -0,0 +1,367 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import (
+ "bytes"
+ "errors"
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/cache/status"
+
+ cr "github.com/Comcast/trickster/internal/cache/registration"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/ranges/byterange"
+ "github.com/Comcast/trickster/internal/proxy/request"
+)
+
+func TestCheckCacheFreshness(t *testing.T) {
+
+ // CachingPolicy should be nil and will return false
+ pr := proxyRequest{}
+ if pr.checkCacheFreshness() {
+ t.Errorf("got %t expected %t", pr.checkCacheFreshness(), false)
+ }
+
+}
+
+func TestParseRequestRanges(t *testing.T) {
+
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ r.Header.Set(headers.NameRange, "bytes=0-10")
+
+ oc := &config.OriginConfig{MultipartRangesDisabled: true}
+ r = request.SetResources(r, request.NewResources(oc, nil, nil, nil, nil))
+
+ pr := proxyRequest{
+ Request: r,
+ upstreamRequest: r,
+ }
+ pr.parseRequestRanges()
+
+ if pr.wantedRanges == nil || len(pr.wantedRanges) < 1 {
+ t.Errorf("unexpected range parse: %v", pr.wantedRanges)
+ }
+
+ r.Header.Set(headers.NameRange, "bytes=0-10,12-20")
+ pr.parseRequestRanges()
+
+ if pr.wantedRanges != nil {
+ t.Errorf("unexpected nil got %s", pr.wantedRanges.String())
+ }
+}
+
+func TestStripConditionalHeaders(t *testing.T) {
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ r.Header.Set(headers.NameIfNoneMatch, "test")
+ pr := proxyRequest{
+ upstreamRequest: r,
+ cachingPolicy: &CachingPolicy{IsClientConditional: true},
+ }
+ pr.stripConditionalHeaders()
+ if v := r.Header.Get(headers.NameIfNoneMatch); v == "test" {
+ t.Errorf("expected header to be stripped: %s", headers.NameIfNoneMatch)
+ }
+}
+
+func TestSetBodyWriter(t *testing.T) {
+
+ buff := make([]byte, 0)
+ pr := proxyRequest{
+ writeToCache: true,
+ contentLength: -1,
+ responseWriter: bytes.NewBuffer(buff),
+ upstreamResponse: &http.Response{StatusCode: http.StatusOK},
+ cachingPolicy: &CachingPolicy{},
+ }
+
+ PrepareResponseWriter(pr.responseWriter, pr.upstreamResponse.StatusCode, pr.upstreamResponse.Header)
+
+ pr.setBodyWriter()
+ if pr.cacheBuffer == nil {
+ t.Error(errors.New("expected non-nil cacheBody"))
+ }
+
+ pr.cachingPolicy.IsClientFresh = true
+ pr.cacheBuffer = nil
+ pr.upstreamResponse.StatusCode = http.StatusNotModified
+
+ pr.setBodyWriter()
+ if pr.cacheBuffer == nil {
+ t.Error(errors.New("expected non-nil cacheBody"))
+ }
+
+}
+
+func TestWriteResponseBody(t *testing.T) {
+
+ pr := proxyRequest{}
+ pr.writeResponseBody()
+ if pr.responseWriter != nil {
+ t.Error(errors.New("expected nil writer"))
+ }
+}
+
+func TestDetermineCacheability(t *testing.T) {
+
+ err := config.Load("trickster", "test", []string{"-origin-url", "http://1", "-origin-type", "test"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1", nil)
+ r = request.SetResources(r, request.NewResources(nil, nil, cache.Configuration(), cache, nil))
+
+ pr := proxyRequest{
+ Request: r,
+ cachingPolicy: &CachingPolicy{NoCache: true, LastModified: time.Unix(1, 0)},
+ writeToCache: true,
+ cacheDocument: &HTTPDocument{
+ CachingPolicy: &CachingPolicy{},
+ },
+ }
+ pr.determineCacheability()
+ if pr.writeToCache {
+ t.Errorf("expected %t got %t", false, pr.writeToCache)
+ }
+
+ pr.revalidation = RevalStatusLocal
+ pr.cacheDocument.CachingPolicy.LastModified = time.Unix(0, 0)
+ pr.cachingPolicy.NoCache = false
+ pr.cachingPolicy.HasIfModifiedSince = true
+ pr.determineCacheability()
+
+ if pr.cacheStatus != status.LookupStatusKeyMiss {
+ t.Errorf("expected %s got %s", status.LookupStatusKeyMiss, pr.cacheStatus)
+ }
+}
+
+func TestStoreNoWrite(t *testing.T) {
+ pr := proxyRequest{}
+ err := pr.store()
+ if err != nil {
+ t.Error(err)
+ }
+}
+
+func TestUpdateContentLengthNilResponse(t *testing.T) {
+ pr := proxyRequest{contentLength: -1}
+ pr.updateContentLength()
+ if pr.contentLength != -1 {
+ t.Errorf("expected %d got %d", -1, pr.contentLength)
+ }
+}
+
+func TestPrepareResponse(t *testing.T) {
+
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ r.Header.Set(headers.NameRange, "bytes=0-10")
+
+ oc := &config.OriginConfig{}
+ r = request.SetResources(r, request.NewResources(oc, nil, nil, nil, nil))
+
+ pr := proxyRequest{
+ Request: r,
+ cachingPolicy: &CachingPolicy{},
+ upstreamResponse: &http.Response{StatusCode: http.StatusOK},
+ cacheDocument: &HTTPDocument{},
+ }
+ pr.parseRequestRanges()
+
+ pr.cacheDocument.Ranges = pr.wantedRanges
+
+ if !pr.wantsRanges || len(pr.wantedRanges) < 1 {
+ t.Errorf("unexpected range parse: %v", pr.wantedRanges)
+ }
+
+ pr.prepareResponse()
+
+ // test again with full body and needed ranges
+ pr.upstreamResponse.StatusCode = http.StatusOK
+ pr.cacheStatus = status.LookupStatusKeyMiss
+ pr.writeToCache = true
+ pr.upstreamReader = bytes.NewBufferString("trickster")
+ headers.Merge(pr.upstreamResponse.Header, http.Header{
+ headers.NameContentRange: {"bytes 0-9"},
+ })
+
+ pr.prepareResponse()
+
+ if pr.upstreamResponse.StatusCode != http.StatusPartialContent {
+ t.Errorf("expected %d got %d", http.StatusPartialContent, pr.upstreamResponse.StatusCode)
+ }
+
+ // test again with no ranges
+ pr.wantsRanges = false
+ pr.wantedRanges = nil
+ pr.prepareResponse()
+
+ //t.Errorf("%s", "foo")
+
+}
+
+func TestPrepareResponsePreconditionFailed(t *testing.T) {
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ pr := proxyRequest{
+ Request: r,
+ cachingPolicy: &CachingPolicy{
+ IsClientConditional: true,
+ IsClientFresh: true,
+ HasIfNoneMatch: true,
+ IfNoneMatchResult: false,
+ ETag: "1234",
+ IfNoneMatchValue: "1234",
+ },
+ upstreamResponse: &http.Response{},
+ cacheDocument: &HTTPDocument{},
+ }
+ pr.Method = http.MethodPost
+ pr.prepareResponse()
+ if pr.upstreamResponse.StatusCode != http.StatusPreconditionFailed {
+ t.Errorf("expected %d got %d", http.StatusPreconditionFailed, pr.upstreamResponse.StatusCode)
+ }
+}
+
+func TestPrepareRevalidationRequest(t *testing.T) {
+
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ r.Header.Set(headers.NameRange, "bytes=0-10,12-20")
+
+ oc := &config.OriginConfig{DearticulateUpstreamRanges: true}
+ r = request.SetResources(r, request.NewResources(oc, nil, nil, nil, nil))
+
+ pr := proxyRequest{
+ Request: r,
+ upstreamRequest: r,
+ cachingPolicy: &CachingPolicy{},
+ upstreamResponse: &http.Response{},
+ cacheDocument: &HTTPDocument{Ranges: byterange.Ranges{byterange.Range{Start: 30, End: 40}}},
+ cacheStatus: status.LookupStatusPartialHit,
+ wantedRanges: byterange.Ranges{{Start: 0, End: 10}, {Start: 12, End: 20}},
+ }
+ pr.prepareRevalidationRequest()
+
+ v := pr.revalidationRequest.Header.Get(headers.NameRange)
+ expected := pr.cacheDocument.Ranges.String()
+
+ if v != expected {
+ t.Errorf("expected %s got %s", expected, v)
+ }
+
+}
+
+func TestPrepareRevalidationRequestNoRange(t *testing.T) {
+
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ r.Header.Set(headers.NameRange, "bytes=0-10,12-20")
+
+ oc := &config.OriginConfig{DearticulateUpstreamRanges: true}
+ r = request.SetResources(r, request.NewResources(oc, nil, nil, nil, nil))
+
+ pr := proxyRequest{
+ Request: r,
+ upstreamRequest: r,
+ cachingPolicy: &CachingPolicy{},
+ upstreamResponse: &http.Response{},
+ cacheDocument: &HTTPDocument{}, //Ranges: byterange.Ranges{byterange.Range{Start: 30, End: 40}}},
+ cacheStatus: status.LookupStatusPartialHit,
+ wantedRanges: byterange.Ranges{{Start: 0, End: 10}, {Start: 12, End: 20}},
+ }
+ pr.prepareRevalidationRequest()
+
+ v := pr.revalidationRequest.Header.Get(headers.NameRange)
+ expected := ""
+
+ if v != expected {
+ t.Errorf("expected %s got %s", expected, v)
+ }
+}
+
+func TestPrepareUpstreamRequests(t *testing.T) {
+
+ r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1/", nil)
+ r.Header.Set(headers.NameRange, "bytes=0-10,12-20")
+
+ oc := &config.OriginConfig{DearticulateUpstreamRanges: true}
+ r = request.SetResources(r, request.NewResources(oc, nil, nil, nil, nil))
+
+ pr := proxyRequest{
+ Request: r,
+ upstreamRequest: r,
+ cachingPolicy: &CachingPolicy{},
+ upstreamResponse: &http.Response{},
+ cacheDocument: &HTTPDocument{Ranges: byterange.Ranges{byterange.Range{Start: 30, End: 40}}},
+ cacheStatus: status.LookupStatusPartialHit,
+ wantedRanges: byterange.Ranges{{Start: 0, End: 10}, {Start: 12, End: 20}},
+ neededRanges: byterange.Ranges{{Start: 0, End: 10}, {Start: 12, End: 20}},
+ }
+
+ pr.prepareUpstreamRequests()
+
+ expected := 2
+ v := len(pr.originRequests)
+
+ if v != expected {
+ t.Errorf("expected %d got %d", expected, v)
+ }
+}
+
+func TestStoreTrueContentType(t *testing.T) {
+
+ ts, _, r, _, _ := setupTestHarnessOPC("", "test", http.StatusOK, nil)
+ defer ts.Close()
+
+ expected := "1234"
+
+ pr := proxyRequest{
+ Request: r,
+ cachingPolicy: &CachingPolicy{NoCache: true, LastModified: time.Unix(1, 0)},
+ writeToCache: true,
+ cacheDocument: &HTTPDocument{
+ CachingPolicy: &CachingPolicy{},
+ ContentType: "5678",
+ },
+ trueContentType: expected,
+ }
+
+ err := pr.store()
+ if err != nil {
+ t.Error(err)
+ }
+
+ if pr.cacheDocument.ContentType != expected {
+ t.Errorf("expected %s got %s", expected, pr.cacheDocument.ContentType)
+ }
+
+}
+
+func TestReconstituteResponses(t *testing.T) {
+
+ pr := &proxyRequest{}
+
+ pr.reconstituteResponses()
+ if len(pr.originRequests) != 0 {
+ t.Errorf("expected %d got %d", 0, len(pr.originRequests))
+ }
+
+}
diff --git a/internal/proxy/engines/revalidation_status.go b/internal/proxy/engines/revalidation_status.go
new file mode 100644
index 000000000..fd5ef2cb6
--- /dev/null
+++ b/internal/proxy/engines/revalidation_status.go
@@ -0,0 +1,59 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import "strconv"
+
+// RevalidationStatus enumerates the methodologies for maintaining time series cache data
+type RevalidationStatus int
+
+const (
+ // RevalStatusNone indicates the object will not undergo revalidation against the origin
+ RevalStatusNone = RevalidationStatus(iota)
+ // RevalStatusInProgress indicates the object is currently being revalidated against the origin
+ RevalStatusInProgress
+ // RevalStatusLocal is used during a cache Range Miss, and indicates that while the user-requested ranges
+ // for an object are uncached and must be fetched from the origin, other ranges for the object are cached
+ // but require revalidation. When a request is in this state, Trickster will use the response headers from
+ // the range miss to locally revalidate the cached content instead of making a separate request to the
+ // origin for revalidating the cached ranges.
+ RevalStatusLocal
+ // RevalStatusOK indicates the object was successfully revalidated against the origin and is still fresh
+ RevalStatusOK
+ // RevalStatusFailed indicates the origin returned a new object for the URL to replace the cached version
+ RevalStatusFailed
+)
+
+var revalidationStatusNames = map[string]RevalidationStatus{
+ "none": RevalStatusNone,
+ "revalidating": RevalStatusInProgress,
+ "revalidated": RevalStatusOK,
+ "failed": RevalStatusFailed,
+ "local": RevalStatusLocal,
+}
+
+var revalidationStatusValues = map[RevalidationStatus]string{
+ RevalStatusNone: "none",
+ RevalStatusInProgress: "revalidating",
+ RevalStatusOK: "revalidated",
+ RevalStatusFailed: "failed",
+ RevalStatusLocal: "local",
+}
+
+func (s RevalidationStatus) String() string {
+ if v, ok := revalidationStatusValues[s]; ok {
+ return v
+ }
+ return strconv.Itoa(int(s))
+}
diff --git a/internal/proxy/engines/revalidation_status_test.go b/internal/proxy/engines/revalidation_status_test.go
new file mode 100644
index 000000000..91f53c0cf
--- /dev/null
+++ b/internal/proxy/engines/revalidation_status_test.go
@@ -0,0 +1,36 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package engines
+
+import "testing"
+
+func TestRevalidationStatusString(t *testing.T) {
+
+ t1 := RevalStatusNone
+ t2 := RevalStatusInProgress
+ var t3 RevalidationStatus = 10
+
+ if t1.String() != "none" {
+ t.Errorf("expected %s got %s", "none", t1.String())
+ }
+
+ if t2.String() != "revalidating" {
+ t.Errorf("expected %s got %s", "revalidating", t2.String())
+ }
+
+ if t3.String() != "10" {
+ t.Errorf("expected %s got %s", "10", t3.String())
+ }
+
+}
diff --git a/internal/proxy/errors/errors.go b/internal/proxy/errors/errors.go
new file mode 100644
index 000000000..4a1ffd9f1
--- /dev/null
+++ b/internal/proxy/errors/errors.go
@@ -0,0 +1,64 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package errors
+
+import (
+ "errors"
+ "fmt"
+ "time"
+)
+
+// ErrStepParse indicates an error parsing the step interval of a time series request
+var ErrStepParse = errors.New("unable to parse timeseries step from downstream request")
+
+// ErrNotSelectStatement indicates an error that the time series request is not a read-only select query
+var ErrNotSelectStatement = errors.New("not a select statement")
+
+// ErrNotTimeRangeQuery indicates an error that the time series request does not contain a query
+var ErrNotTimeRangeQuery = errors.New("not a time range query")
+
+// ErrNoRanges indicates an error that the range request does not contain any usable ranges
+var ErrNoRanges = errors.New("no usable ranges")
+
+// MissingURLParam returns a Formatted Error
+func MissingURLParam(param string) error {
+ return fmt.Errorf("missing URL parameter: [%s]", param)
+}
+
+// TimeArrayEmpty returns a Formatted Error
+func TimeArrayEmpty(param string) error {
+ return fmt.Errorf("time array is nil or empty: [%s]", param)
+}
+
+// InvalidPath returns an error indicating the request path is not valid.
+func InvalidPath(path string) error {
+ return fmt.Errorf("invalid request path: %s", path)
+}
+
+// ParseDuration returns a Duration Parsing Error
+func ParseDuration(input string) (time.Duration, error) {
+ return time.Duration(0), fmt.Errorf("unable to parse duration: %s", input)
+}
+
+// ParseRequestBody returns an error indicating the request body could not
+// parsed into a valid value.
+func ParseRequestBody(err error) error {
+ return fmt.Errorf("unable to parse request body: %v", err)
+}
+
+// MissingRequestParam returns an error indicating the request is missing a
+// required parameter.
+func MissingRequestParam(param string) error {
+ return fmt.Errorf("missing request parameter: %s", param)
+}
diff --git a/internal/proxy/errors/errors_test.go b/internal/proxy/errors/errors_test.go
new file mode 100644
index 000000000..ffc1bee6c
--- /dev/null
+++ b/internal/proxy/errors/errors_test.go
@@ -0,0 +1,59 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package errors
+
+import (
+ "fmt"
+ "testing"
+)
+
+func TestMissingURLParam(t *testing.T) {
+ if MissingURLParam("test").Error() != "missing URL parameter: [test]" {
+ t.Errorf("ErrorStepParse failed")
+ }
+}
+
+func TestTimeArrayEmpty(t *testing.T) {
+ if TimeArrayEmpty("test").Error() != "time array is nil or empty: [test]" {
+ t.Errorf("ErrorStepParse failed")
+ }
+}
+
+func TestParseDurationError(t *testing.T) {
+ _, err := ParseDuration("test")
+ if err.Error() != "unable to parse duration: test" {
+ t.Errorf("ErrorParseDuration failed")
+ }
+}
+
+func TestInvalidPath(t *testing.T) {
+ err := InvalidPath("test")
+ if err.Error() != "invalid request path: test" {
+ t.Errorf("ErrorInvalidPath failed, got: %v", err.Error())
+ }
+}
+
+func TestParseRequestBody(t *testing.T) {
+ err := ParseRequestBody(fmt.Errorf("test"))
+ if err.Error() != "unable to parse request body: test" {
+ t.Errorf("ErrorParseDuration failed, got: %v", err.Error())
+ }
+}
+
+func TestMissingRequestParam(t *testing.T) {
+ err := MissingRequestParam("test")
+ if err.Error() != "missing request parameter: test" {
+ t.Errorf("ErrorMissingRequestParam failed, got: %v", err.Error())
+ }
+}
diff --git a/internal/proxy/handlers/config.go b/internal/proxy/handlers/config.go
new file mode 100644
index 000000000..00e4becb8
--- /dev/null
+++ b/internal/proxy/handlers/config.go
@@ -0,0 +1,35 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package handlers
+
+import (
+ "net/http"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/routing"
+)
+
+// RegisterConfigHandler registers the application's /ping handler
+func RegisterConfigHandler() {
+ routing.Router.HandleFunc(config.Main.ConfigHandlerPath, configHandler).Methods("GET")
+}
+
+// configHandler responds to an HTTP Request with 200 OK and "pong"
+func configHandler(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set(headers.NameContentType, headers.ValueTextPlain)
+ w.Header().Set(headers.NameCacheControl, headers.ValueNoCache)
+ w.WriteHeader(http.StatusOK)
+ w.Write([]byte(config.Config.String()))
+}
diff --git a/internal/proxy/handlers/config_test.go b/internal/proxy/handlers/config_test.go
new file mode 100644
index 000000000..c26c00d0d
--- /dev/null
+++ b/internal/proxy/handlers/config_test.go
@@ -0,0 +1,54 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package handlers
+
+import (
+ "io/ioutil"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/config"
+)
+
+func TestConfigHandler(t *testing.T) {
+
+ config.Load("trickster-test", "test", []string{"-origin-url", "http://1.2.3.4", "-origin-type", "prometheus"})
+
+ RegisterConfigHandler()
+
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", "http://0/trickster/config", nil)
+
+ configHandler(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK and "pong"
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if len(bodyBytes) < 1 {
+ t.Errorf("missing body in response")
+ }
+
+ if bodyBytes[0] != 91 {
+ t.Errorf("response is not toml format")
+ }
+
+}
diff --git a/internal/proxy/handlers/local.go b/internal/proxy/handlers/local.go
new file mode 100644
index 000000000..777683393
--- /dev/null
+++ b/internal/proxy/handlers/local.go
@@ -0,0 +1,39 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package handlers
+
+import (
+ "net/http"
+
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+)
+
+// HandleLocalResponse responds to an HTTP Request based on the local configuration without making any upstream requests
+func HandleLocalResponse(w http.ResponseWriter, r *http.Request) {
+ rsc := request.GetResources(r)
+ p := rsc.PathConfig
+ if p == nil {
+ return
+ }
+ if len(p.ResponseHeaders) > 0 {
+ headers.UpdateHeaders(w.Header(), p.ResponseHeaders)
+ }
+ if p.ResponseCode > 0 {
+ w.WriteHeader(p.ResponseCode)
+ } else {
+ w.WriteHeader(http.StatusOK)
+ }
+ w.Write([]byte(p.ResponseBody))
+}
diff --git a/internal/proxy/handlers/local_test.go b/internal/proxy/handlers/local_test.go
new file mode 100644
index 000000000..8a1e5b48b
--- /dev/null
+++ b/internal/proxy/handlers/local_test.go
@@ -0,0 +1,139 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package handlers
+
+import (
+ "io/ioutil"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/config"
+ tc "github.com/Comcast/trickster/internal/proxy/context"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/proxy/request"
+)
+
+func TestHandleLocalResponse(t *testing.T) {
+
+ config.Load("trickster-test", "test", []string{"-origin-url", "http://1.2.3.4", "-origin-type", "prometheus"})
+
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", "http://0/trickster/", nil)
+
+ pc := &config.PathConfig{
+ ResponseCode: 418,
+ ResponseBody: "[test",
+ ResponseBodyBytes: []byte("[test"),
+ ResponseHeaders: map[string]string{headers.NameTricksterResult: "1234"},
+ }
+
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(nil, pc, nil, nil, nil)))
+
+ HandleLocalResponse(w, r)
+ resp := w.Result()
+
+ // it should return 418 OK and "pong"
+ if resp.StatusCode != 418 {
+ t.Errorf("expected 418 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if len(bodyBytes) < 1 {
+ t.Errorf("missing body in response")
+ }
+
+ if bodyBytes[0] != 91 {
+ t.Errorf("response is not toml format")
+ }
+
+ if resp.Header.Get(headers.NameTricksterResult) == "" {
+ t.Errorf("expected header valuef for %s", headers.NameTricksterResult)
+ }
+
+}
+
+func TestHandleLocalResponseBadResponseCode(t *testing.T) {
+
+ config.Load("trickster-test", "test", []string{"-origin-url", "http://1.2.3.4", "-origin-type", "prometheus"})
+
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", "http://0/trickster/", nil)
+
+ pc := &config.PathConfig{
+ ResponseCode: 0,
+ ResponseBody: "[test",
+ ResponseBodyBytes: []byte("[test"),
+ ResponseHeaders: map[string]string{headers.NameTricksterResult: "1234"},
+ }
+
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(nil, pc, nil, nil, nil)))
+
+ HandleLocalResponse(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK and because we passed 0
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if len(bodyBytes) < 1 {
+ t.Errorf("missing body in response")
+ }
+
+ if bodyBytes[0] != 91 {
+ t.Errorf("response is not toml format")
+ }
+
+ if resp.Header.Get(headers.NameTricksterResult) == "" {
+ t.Errorf("expected header valuef for %s", headers.NameTricksterResult)
+ }
+
+}
+
+func TestHandleLocalResponseNoPathConfig(t *testing.T) {
+
+ config.Load("trickster-test", "test", []string{"-origin-url", "http://1.2.3.4", "-origin-type", "prometheus"})
+
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", "http://0/trickster/", nil)
+
+ r = r.WithContext(tc.WithResources(r.Context(), request.NewResources(nil, nil, nil, nil, nil)))
+
+ HandleLocalResponse(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK and "pong"
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if len(bodyBytes) > 0 {
+ t.Errorf("body should be empty")
+ }
+
+}
diff --git a/internal/proxy/handlers/ping.go b/internal/proxy/handlers/ping.go
new file mode 100644
index 000000000..d795c46e5
--- /dev/null
+++ b/internal/proxy/handlers/ping.go
@@ -0,0 +1,35 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package handlers
+
+import (
+ "net/http"
+
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+ "github.com/Comcast/trickster/internal/routing"
+)
+
+// RegisterPingHandler registers the application's /ping handler
+func RegisterPingHandler() {
+ routing.Router.HandleFunc(config.Main.PingHandlerPath, pingHandler).Methods("GET")
+}
+
+// pingHandler responds to an HTTP Request with 200 OK and "pong"
+func pingHandler(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set(headers.NameContentType, headers.ValueTextPlain)
+ w.Header().Set(headers.NameCacheControl, headers.ValueNoCache)
+ w.WriteHeader(http.StatusOK)
+ w.Write([]byte("pong"))
+}
diff --git a/internal/proxy/handlers/ping_test.go b/internal/proxy/handlers/ping_test.go
new file mode 100644
index 000000000..d68fa3c6d
--- /dev/null
+++ b/internal/proxy/handlers/ping_test.go
@@ -0,0 +1,49 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package handlers
+
+import (
+ "io/ioutil"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/config"
+)
+
+func TestPingHandler(t *testing.T) {
+
+ config.Load("trickster-test", "test", nil)
+ RegisterPingHandler()
+
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest("GET", "http://0/trickster/ping", nil)
+
+ pingHandler(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK and "pong"
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(bodyBytes) != "pong" {
+ t.Errorf("expected 'pong' got %s.", bodyBytes)
+ }
+
+}
diff --git a/internal/proxy/headers/headers.go b/internal/proxy/headers/headers.go
new file mode 100644
index 000000000..df30991c4
--- /dev/null
+++ b/internal/proxy/headers/headers.go
@@ -0,0 +1,218 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package headers
+
+import (
+ "fmt"
+ "net/http"
+ "strings"
+
+ "github.com/Comcast/trickster/internal/runtime"
+ "github.com/Comcast/trickster/internal/timeseries"
+)
+
+const (
+ // Common HTTP Header Values
+
+ // ValueApplicationJSON represents the HTTP Header Value of "application/json"
+ ValueApplicationJSON = "application/json"
+ // ValueMaxAge represents the HTTP Header Value of "max-age"
+ ValueMaxAge = "max-age"
+ // ValueMultipartFormData represents the HTTP Header Value of "multipart/form-data"
+ ValueMultipartFormData = "multipart/form-data"
+ // ValueMustRevalidate represents the HTTP Header Value of "must-revalidate"
+ ValueMustRevalidate = "must-revalidate"
+ // ValueNoCache represents the HTTP Header Value of "no-cache"
+ ValueNoCache = "no-cache"
+ // ValueNoStore represents the HTTP Header Value of "no-store"
+ ValueNoStore = "no-store"
+ // ValueNoTransform represents the HTTP Header Value of "no-transform"
+ ValueNoTransform = "no-transform"
+ // ValuePrivate represents the HTTP Header Value of "private"
+ ValuePrivate = "private"
+ // ValueProxyRevalidate represents the HTTP Header Value of "proxy-revalidate"
+ ValueProxyRevalidate = "proxy-revalidate"
+ // ValuePublic represents the HTTP Header Value of "public"
+ ValuePublic = "public"
+ // ValueSharedMaxAge represents the HTTP Header Value of "s-maxage"
+ ValueSharedMaxAge = "s-maxage"
+ // ValueTextPlain represents the HTTP Header Value of "text/plain"
+ ValueTextPlain = "text/plain"
+ // ValueXFormURLEncoded represents the HTTP Header Value of "application/x-www-form-urlencoded"
+ ValueXFormURLEncoded = "application/x-www-form-urlencoded"
+
+ // ValueMultipartByteRanges represents the HTTP Header prefix for a Multipart Byte Range response
+ ValueMultipartByteRanges = "multipart/byteranges; boundary="
+
+ // Common HTTP Header Names
+
+ // NameCacheControl represents the HTTP Header Name of "Cache-Control"
+ NameCacheControl = "Cache-Control"
+ // NameAllowOrigin represents the HTTP Header Name of "Access-Control-Allow-Origin"
+ NameAllowOrigin = "Access-Control-Allow-Origin"
+ // NameContentType represents the HTTP Header Name of "Content-Type"
+ NameContentType = "Content-Type"
+ // NameContentEncoding represents the HTTP Header Name of "Content-Encoding"
+ NameContentEncoding = "Content-Encoding"
+ // NameContentLength represents the HTTP Header Name of "Content-Length"
+ NameContentLength = "Content-Length"
+ // NameAuthorization represents the HTTP Header Name of "Authorization"
+ NameAuthorization = "Authorization"
+ // NameContentRange represents the HTTP Header Name of "Content-Range"
+ NameContentRange = "Content-Range"
+ // NameTricksterResult represents the HTTP Header Name of "X-Trickster-Result"
+ NameTricksterResult = "X-Trickster-Result"
+ // NameVia represents the HTTP Header Name of "Via"
+ NameVia = "Via"
+ // NameXForwardedFor represents the HTTP Header Name of "X-Forwarded-For"
+ NameXForwardedFor = "X-Forwarded-For"
+ // NameAcceptEncoding represents the HTTP Header Name of "Accept-Encoding"
+ NameAcceptEncoding = "Accept-Encoding"
+ // NameSetCookie represents the HTTP Header Name of "Set-Cookie"
+ NameSetCookie = "Set-Cookie"
+ // NameRange represents the HTTP Header Name of "Range"
+ NameRange = "Range"
+ // NameTransferEncoding represents the HTTP Header Name of "Transfer-Encoding"
+ NameTransferEncoding = "Transfer-Encoding"
+ // NameIfModifiedSince represents the HTTP Header Name of "If-Modified-Since"
+ NameIfModifiedSince = "If-Modified-Since"
+ // NameIfUnmodifiedSince represents the HTTP Header Name of "If-Unodified-Since"
+ NameIfUnmodifiedSince = "If-Unmodified-Since"
+ // NameIfNoneMatch represents the HTTP Header Name of "If-None-Match"
+ NameIfNoneMatch = "If-None-Match"
+ // NameIfMatch represents the HTTP Header Name of "If-Match"
+ NameIfMatch = "If-Match"
+ // NameDate represents the HTTP Header Name of "date"
+ NameDate = "Date"
+ // NamePragma represents the HTTP Header Name of "pragma"
+ NamePragma = "Pragma"
+ // NameLastModified represents the HTTP Header Name of "last-modified"
+ NameLastModified = "Last-Modified"
+ // NameExpires represents the HTTP Header Name of "expires"
+ NameExpires = "Expires"
+ // NameETag represents the HTTP Header Name of "etag"
+ NameETag = "Etag"
+)
+
+// Merge merges the source http.Header map into destination map.
+// If a key exists in both maps, the source value wins.
+// If the destination map is nil, the source map will not be merged
+func Merge(dst, src http.Header) {
+ if src == nil || len(src) == 0 || dst == nil {
+ return
+ }
+ for k, sv := range src {
+ if len(sv) == 0 {
+ continue
+ }
+ dst[k] = []string{sv[0]}
+ }
+}
+
+// UpdateHeaders updates the provided headers collection with the provided updates
+func UpdateHeaders(headers http.Header, updates map[string]string) {
+ if headers == nil || updates == nil || len(updates) == 0 {
+ return
+ }
+ for k, v := range updates {
+ if len(k) == 0 {
+ continue
+ }
+ if k[0:1] == "-" {
+ k = k[1:]
+ headers.Del(k)
+ continue
+ }
+ if k[0:1] == "+" {
+ k = k[1:]
+ headers.Add(k, v)
+ continue
+ }
+ headers.Set(k, v)
+ }
+}
+
+// AddProxyHeaders injects standard Trickster headers into proxied upstream HTTP requests
+func AddProxyHeaders(remoteAddr string, headers http.Header) {
+ if remoteAddr != "" {
+ headers.Set(NameXForwardedFor, remoteAddr)
+ headers.Set(NameVia, runtime.ApplicationName+" "+runtime.ApplicationVersion)
+ }
+}
+
+// AddResponseHeaders injects standard Trickster headers into downstream HTTP responses
+func AddResponseHeaders(headers http.Header) {
+ // We're read only and a harmless API, so allow all CORS
+ headers.Set(NameAllowOrigin, "*")
+ headers.Set(NameVia, runtime.ApplicationName+" "+runtime.ApplicationVersion)
+}
+
+// SetResultsHeader adds a response header summarizing Trickster's handling of the HTTP request
+func SetResultsHeader(headers http.Header, engine, status, ffstatus string, fetched timeseries.ExtentList) {
+
+ if headers == nil || engine == "" {
+ return
+ }
+
+ parts := append(make([]string, 0, 4), fmt.Sprintf("engine=%s", engine))
+
+ if status != "" {
+ parts = append(parts, fmt.Sprintf("status=%s", status))
+ }
+
+ if fetched != nil && len(fetched) > 0 {
+ fp := make([]string, 0, len(fetched))
+ for _, v := range fetched {
+ fp = append(fp, fmt.Sprintf("%d:%d", v.Start.Unix(), v.End.Unix()))
+ }
+ parts = append(parts, fmt.Sprintf("fetched=[%s]", strings.Join(fp, ",")))
+ }
+
+ if ffstatus != "" {
+ parts = append(parts, fmt.Sprintf("ffstatus=%s", ffstatus))
+ }
+
+ headers.Set(NameTricksterResult, strings.Join(parts, "; "))
+
+}
+
+// ExtractHeader returns the value for the provided header name, and a boolean indicating if the header was present
+func ExtractHeader(headers http.Header, header string) (string, bool) {
+ if Value, ok := headers[header]; ok {
+ return strings.Join(Value, "; "), true
+ }
+ return "", false
+}
+
+// RemoveClientHeaders strips certain headers from the HTTP request to facililate acceleration
+func RemoveClientHeaders(headers http.Header) {
+ headers.Del(NameAcceptEncoding)
+}
+
+// String returns the string representation of the headers as if
+// they were transmitted over the wire (Header1: value1\nHeader2: value2\n\n)
+func String(h http.Header) string {
+ if h == nil || len(h) == 0 {
+ return "\n\n"
+ }
+ sb := strings.Builder{}
+ for k, v := range h {
+ if len(v) > 0 {
+ sb.WriteString(fmt.Sprintf("%s: %s\n", k, v[0]))
+ }
+ }
+ // add the header section end new line
+ sb.WriteString("\n")
+ return sb.String()
+}
diff --git a/internal/proxy/headers/headers_test.go b/internal/proxy/headers/headers_test.go
new file mode 100644
index 000000000..ffcdb3fba
--- /dev/null
+++ b/internal/proxy/headers/headers_test.go
@@ -0,0 +1,187 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package headers
+
+import (
+ "fmt"
+ "net/http"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/runtime"
+ "github.com/Comcast/trickster/internal/timeseries"
+)
+
+func TestAddProxyHeaders(t *testing.T) {
+
+ headers := http.Header{}
+ runtime.ApplicationName = "trickster-test"
+ runtime.ApplicationVersion = "tests"
+
+ AddProxyHeaders("0.0.0.0", headers)
+
+ if _, ok := headers[NameXForwardedFor]; !ok {
+ t.Errorf("missing header %s", NameXForwardedFor)
+ }
+
+ if _, ok := headers[NameVia]; !ok {
+ t.Errorf("missing header %s", NameVia)
+ }
+
+}
+
+func TestExtractHeader(t *testing.T) {
+
+ headers := http.Header{}
+
+ const appName = "trickster-test"
+ const appVer = "tests"
+ const appString = appName + " " + appVer
+
+ runtime.ApplicationName = appName
+ runtime.ApplicationVersion = appVer
+
+ const testIP = "0.0.0.0"
+
+ AddProxyHeaders(testIP, headers)
+
+ if h, ok := ExtractHeader(headers, NameXForwardedFor); !ok {
+ t.Errorf("missing header %s", NameXForwardedFor)
+ } else {
+ if h != testIP {
+ t.Errorf(`expected "%s". got "%s"`, testIP, h)
+ }
+ }
+
+ if h, ok := ExtractHeader(headers, NameVia); !ok {
+ t.Errorf("missing header %s", NameVia)
+ } else {
+ if h != appString {
+ t.Errorf(`expected "%s". got "%s"`, appString, h)
+ }
+ }
+
+ if _, ok := ExtractHeader(headers, NameAllowOrigin); ok {
+ t.Errorf("unexpected header %s", NameAllowOrigin)
+ }
+
+}
+
+func TestUpdateHeaders(t *testing.T) {
+ headers := http.Header{"Foo1": {"foo"}, "Foo2": {"x"}, "Foo3": {"foo"}}
+ expected := http.Header{"Foo1": {"bar"}, "Foo3": {"foo", "bar"}, "Foo4": {"bar"}, "Foo5": {"bar"}}
+
+ UpdateHeaders(headers, nil)
+ if len(headers) != 3 {
+ t.Errorf("expected %d got %d", len(headers), 3)
+ }
+
+ UpdateHeaders(headers, map[string]string{"": "ineffectual", "foo1": "bar", "-foo2": "", "+foo3": "bar", "foo4": "bar", "+foo5": "bar", "-foo6": ""})
+ if !reflect.DeepEqual(headers, expected) {
+ fmt.Printf("mismatch\nexpected: %v\n got: %v\n", expected, headers)
+ }
+
+}
+
+func TestRemoveClientHeaders(t *testing.T) {
+
+ headers := http.Header{}
+ headers.Set(NameAcceptEncoding, "test")
+
+ RemoveClientHeaders(headers)
+
+ if _, ok := ExtractHeader(headers, NameAcceptEncoding); ok {
+ t.Errorf("unexpected header %s", NameAcceptEncoding)
+ }
+
+}
+
+func TestMerge(t *testing.T) {
+ h1 := make(http.Header)
+ h1.Set("test", "pass")
+ h2 := make(http.Header)
+ h2.Set("test2", "pass")
+
+ Merge(h2, h1)
+ if h2.Get("test") != "pass" {
+ t.Errorf("expected 'pass' got '%s'", h2.Get("test"))
+ }
+
+ Merge(h2, nil)
+ if h2.Get("test") != "pass" {
+ t.Errorf("expected 'pass' got '%s'", h2.Get("test"))
+ }
+
+ h2["test2"] = make([]string, 0)
+
+ Merge(h1, h2)
+ if h1.Get("test") != "pass" {
+ t.Errorf("expected 'pass' got '%s'", h1.Get("test"))
+ }
+
+}
+
+func TestAddResponseHeaders(t *testing.T) {
+
+ headers := http.Header{}
+ runtime.ApplicationName = "trickster-test"
+ runtime.ApplicationVersion = "tests"
+
+ AddResponseHeaders(headers)
+
+ if _, ok := headers[NameAllowOrigin]; !ok {
+ t.Errorf("missing header %s", NameAllowOrigin)
+ }
+
+ if _, ok := headers[NameVia]; !ok {
+ t.Errorf("missing header %s", NameVia)
+ }
+
+}
+
+func TestSetResultsHeader(t *testing.T) {
+ h := http.Header{}
+ SetResultsHeader(h, "test-engine", "test-status", "test-ffstatus", timeseries.ExtentList{timeseries.Extent{Start: time.Unix(1, 0), End: time.Unix(2, 0)}})
+ const expected = "engine=test-engine; status=test-status; fetched=[1:2]; ffstatus=test-ffstatus"
+ if h.Get(NameTricksterResult) != expected {
+ t.Errorf("expected %s got %s", expected, h.Get(NameTricksterResult))
+ }
+}
+
+func TestSetResultsHeaderEmtpy(t *testing.T) {
+ h := http.Header{}
+ SetResultsHeader(h, "", "test-status", "test-ffstatus", timeseries.ExtentList{timeseries.Extent{Start: time.Unix(1, 0), End: time.Unix(2, 0)}})
+ if len(h) > 0 {
+ t.Errorf("Expected header length of %d", 0)
+ }
+}
+
+func TestString(t *testing.T) {
+
+ expected := "test: test\n\n"
+ h := http.Header{"test": {"test"}}
+ x := String(h)
+ if x != expected {
+ t.Errorf("expected %s got %s", expected, x)
+ }
+
+ expected = "\n\n"
+ h = http.Header{}
+ x = String(h)
+ if x != expected {
+ t.Errorf("expected %s got %s", expected, x)
+ }
+
+}
diff --git a/internal/proxy/methods/methods.go b/internal/proxy/methods/methods.go
new file mode 100644
index 000000000..af98dcf19
--- /dev/null
+++ b/internal/proxy/methods/methods.go
@@ -0,0 +1,41 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package methods
+
+import "net/http"
+
+const (
+
+ // Methods not currently in the base golang http package
+
+ // MethodPurge is the PURGE HTTP Method
+ MethodPurge = "PURGE"
+)
+
+// AllHTTPMethods returns a list of all known HTTP methods
+func AllHTTPMethods() []string {
+ return []string{http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut, http.MethodDelete,
+ http.MethodConnect, http.MethodOptions, http.MethodTrace, http.MethodPatch, MethodPurge}
+}
+
+// CacheableHTTPMethods returns a list of HTTP methods that are generally considered cacheable
+func CacheableHTTPMethods() []string {
+ return []string{http.MethodGet, http.MethodHead}
+}
+
+// UncacheableHTTPMethods returns a list of HTTP methods that are generally considered uncacheable
+func UncacheableHTTPMethods() []string {
+ return []string{http.MethodPost, http.MethodPut, http.MethodDelete, http.MethodConnect,
+ http.MethodOptions, http.MethodTrace, http.MethodPatch, MethodPurge}
+}
diff --git a/internal/proxy/methods/methods_test.go b/internal/proxy/methods/methods_test.go
new file mode 100644
index 000000000..9b71441fc
--- /dev/null
+++ b/internal/proxy/methods/methods_test.go
@@ -0,0 +1,42 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package methods
+
+import (
+ "testing"
+)
+
+func TestAllHTTPMethods(t *testing.T) {
+ expected := 10
+ l := len(AllHTTPMethods())
+ if l != expected {
+ t.Errorf("expected %d got %d", expected, l)
+ }
+}
+
+func TestCacheableHTTPMethods(t *testing.T) {
+ expected := 2
+ l := len(CacheableHTTPMethods())
+ if l != expected {
+ t.Errorf("expected %d got %d", expected, l)
+ }
+}
+
+func TestUncacheableHTTPMethods(t *testing.T) {
+ expected := 8
+ l := len(UncacheableHTTPMethods())
+ if l != expected {
+ t.Errorf("expected %d got %d", expected, l)
+ }
+}
diff --git a/internal/proxy/origins/clickhouse/clickhouse.go b/internal/proxy/origins/clickhouse/clickhouse.go
new file mode 100644
index 000000000..5e6a186ef
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/clickhouse.go
@@ -0,0 +1,113 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+ "net/url"
+
+ "github.com/Comcast/trickster/internal/cache"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy"
+ "github.com/Comcast/trickster/internal/proxy/errors"
+ tt "github.com/Comcast/trickster/internal/proxy/timeconv"
+ "github.com/Comcast/trickster/internal/proxy/urls"
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/internal/util/regexp/matching"
+)
+
+// Client Implements the Proxy Client Interface
+type Client struct {
+ name string
+ config *config.OriginConfig
+ cache cache.Cache
+ webClient *http.Client
+ handlers map[string]http.Handler
+ handlersRegistered bool
+
+ healthURL *url.URL
+ healthMethod string
+ healthHeaders http.Header
+}
+
+// NewClient returns a new Client Instance
+func NewClient(name string, oc *config.OriginConfig, cache cache.Cache) (*Client, error) {
+ c, err := proxy.NewHTTPClient(oc)
+ return &Client{name: name, config: oc, cache: cache, webClient: c}, err
+}
+
+// Configuration returns the upstream Configuration for this Client
+func (c *Client) Configuration() *config.OriginConfig {
+ return c.config
+}
+
+// HTTPClient returns the HTTP Transport the client is using
+func (c *Client) HTTPClient() *http.Client {
+ return c.webClient
+}
+
+// Cache returns and handle to the Cache instance used by the Client
+func (c *Client) Cache() cache.Cache {
+ return c.cache
+}
+
+// Name returns the name of the upstream Configuration proxied by the Client
+func (c *Client) Name() string {
+ return c.name
+}
+
+// SetCache sets the Cache object the client will use for caching origin content
+func (c *Client) SetCache(cc cache.Cache) {
+ c.cache = cc
+}
+
+// ParseTimeRangeQuery parses the key parts of a TimeRangeQuery from the inbound HTTP Request
+func (c *Client) ParseTimeRangeQuery(r *http.Request) (*timeseries.TimeRangeQuery, error) {
+
+ trq := ×eries.TimeRangeQuery{Extent: timeseries.Extent{}}
+ trq.TemplateURL = urls.Clone(r.URL)
+ qi := trq.TemplateURL.Query()
+ if p, ok := qi[upQuery]; ok {
+ trq.Statement = p[0]
+ } else {
+ return nil, errors.MissingURLParam(upQuery)
+ }
+
+ mp := []string{"step", "timeField"}
+ found := matching.GetNamedMatches(reTimeFieldAndStep, trq.Statement, mp)
+
+ for _, f := range mp {
+ v, ok := found[f]
+ if !ok || v == "" {
+ return nil, errors.ErrNotTimeRangeQuery
+ }
+ switch f {
+ case "timeField":
+ trq.TimestampFieldName = v
+ case "step":
+ trq.Step, _ = tt.ParseDuration(v + "s")
+ }
+ }
+
+ var err error
+ trq.Statement, trq.Extent, _, err = getQueryParts(trq.Statement, trq.TimestampFieldName)
+ if err != nil {
+ return nil, err
+ }
+
+ // Swap in the Tokenzed Query in the Url Params
+ qi.Set(upQuery, trq.Statement)
+ trq.TemplateURL.RawQuery = qi.Encode()
+ return trq, nil
+}
diff --git a/internal/proxy/origins/clickhouse/clickhouse_test.go b/internal/proxy/origins/clickhouse/clickhouse_test.go
new file mode 100644
index 000000000..46b0d4bed
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/clickhouse_test.go
@@ -0,0 +1,193 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+ "net/url"
+ "testing"
+
+ cr "github.com/Comcast/trickster/internal/cache/registration"
+ "github.com/Comcast/trickster/internal/config"
+ "github.com/Comcast/trickster/internal/proxy/origins"
+ "github.com/Comcast/trickster/internal/util/metrics"
+)
+
+func init() {
+ metrics.Init()
+}
+
+func TestClickhouseClientInterfacing(t *testing.T) {
+
+ // this test ensures the client will properly conform to the
+ // Client and TimeseriesClient interfaces
+
+ c := &Client{name: "test"}
+ var oc origins.Client = c
+ var tc origins.TimeseriesClient = c
+
+ if oc.Name() != "test" {
+ t.Errorf("expected %s got %s", "test", oc.Name())
+ }
+
+ if tc.Name() != "test" {
+ t.Errorf("expected %s got %s", "test", tc.Name())
+ }
+}
+
+func TestNewClient(t *testing.T) {
+
+ err := config.Load("trickster", "test", []string{"-origin-type", "clickhouse", "-origin-url", "http://1"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+
+ oc := &config.OriginConfig{OriginType: "TEST_CLIENT"}
+ c, err := NewClient("default", oc, cache)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if c.Name() != "default" {
+ t.Errorf("expected %s got %s", "default", c.Name())
+ }
+
+ if c.Cache().Configuration().CacheType != "memory" {
+ t.Errorf("expected %s got %s", "memory", c.Cache().Configuration().CacheType)
+ }
+
+ if c.Configuration().OriginType != "TEST_CLIENT" {
+ t.Errorf("expected %s got %s", "TEST_CLIENT", c.Configuration().OriginType)
+ }
+}
+
+func TestConfiguration(t *testing.T) {
+ oc := &config.OriginConfig{OriginType: "TEST"}
+ client := Client{config: oc}
+ c := client.Configuration()
+ if c.OriginType != "TEST" {
+ t.Errorf("expected %s got %s", "TEST", c.OriginType)
+ }
+}
+
+func TestCache(t *testing.T) {
+
+ err := config.Load("trickster", "test", []string{"-origin-type", "clickhouse", "-origin-url", "http://1"})
+ if err != nil {
+ t.Errorf("Could not load configuration: %s", err.Error())
+ }
+
+ cr.LoadCachesFromConfig()
+ cache, err := cr.GetCache("default")
+ if err != nil {
+ t.Error(err)
+ }
+ client := Client{cache: cache}
+ c := client.Cache()
+
+ if c.Configuration().CacheType != "memory" {
+ t.Errorf("expected %s got %s", "memory", c.Configuration().CacheType)
+ }
+}
+
+func TestName(t *testing.T) {
+
+ client := Client{name: "TEST"}
+ c := client.Name()
+
+ if c != "TEST" {
+ t.Errorf("expected %s got %s", "TEST", c)
+ }
+
+}
+
+func TestHTTPClient(t *testing.T) {
+ oc := &config.OriginConfig{OriginType: "TEST"}
+
+ client, err := NewClient("test", oc, nil)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if client.HTTPClient() == nil {
+ t.Errorf("missing http client")
+ }
+}
+
+func TestSetCache(t *testing.T) {
+ c, err := NewClient("test", config.NewOriginConfig(), nil)
+ if err != nil {
+ t.Error(err)
+ }
+ c.SetCache(nil)
+ if c.Cache() != nil {
+ t.Errorf("expected nil cache for client named %s", "test")
+ }
+}
+
+func TestParseTimeRangeQuery(t *testing.T) {
+ req := &http.Request{URL: &url.URL{
+ Scheme: "https",
+ Host: "blah.com",
+ Path: "/",
+ RawQuery: testRawQuery(),
+ }}
+ client := &Client{}
+ res, err := client.ParseTimeRangeQuery(req)
+ if err != nil {
+ t.Error(err)
+ } else {
+
+ if res.Step.Seconds() != 60 {
+ t.Errorf("expected 60 got %f", res.Step.Seconds())
+ }
+
+ if res.Extent.End.Sub(res.Extent.Start).Hours() != 6 {
+ t.Errorf("expected 6 got %f", res.Extent.End.Sub(res.Extent.Start).Hours())
+ }
+ }
+
+ req.URL.RawQuery = ""
+ _, err = client.ParseTimeRangeQuery(req)
+ if err == nil {
+ t.Errorf("expected error for: %s", "missing URL parameter: [query]")
+ }
+
+ req.URL.RawQuery = url.Values(map[string][]string{"query": {
+ `SELECT (intDiv(toUInt32(abc), 6z0) * 6z0) * 1000 AS t, countMerge(some_count) AS cnt, field1, field2 ` +
+ `FROM testdb.test_table WHERE abc BETWEEN toDateTime(1516665600) AND toDateTime(1516687200) ` +
+ `AND date_column >= toDate(1516665600) AND toDate(1516687200) ` +
+ `AND field1 > 0 AND field2 = 'some_value' GROUP BY t, field1, field2 ORDER BY t, field1 FORMAT JSON`}}).Encode()
+ _, err = client.ParseTimeRangeQuery(req)
+ if err == nil {
+ t.Errorf("expected error for: %s", "not a time range query")
+ }
+
+ req.URL.RawQuery = url.Values(map[string][]string{"query": {
+ `SELECT (intDiv(toUInt32(0^^^), 60) * 60) * 1000 AS t, countMerge(some_count) AS cnt, field1, field2 ` +
+ `FROM testdb.test_table WHERE 0^^^ BETWEEN toDateTime(1516665600) AND toDateTime(1516687200) ` +
+ `AND date_column >= toDate(1516665600) AND toDate(1516687200) ` +
+ `AND field1 > 0 AND field2 = 'some_value' GROUP BY t, field1, field2 ORDER BY t, field1 FORMAT JSON`}}).Encode()
+ _, err = client.ParseTimeRangeQuery(req)
+ if err == nil {
+ t.Errorf("expected error for: %s", "not a time range query")
+ }
+
+}
diff --git a/internal/proxy/origins/clickhouse/handler_health.go b/internal/proxy/origins/clickhouse/handler_health.go
new file mode 100644
index 000000000..47975948c
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/handler_health.go
@@ -0,0 +1,73 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+ "net/url"
+
+ "github.com/Comcast/trickster/internal/proxy/engines"
+ "github.com/Comcast/trickster/internal/proxy/headers"
+)
+
+const (
+ healthQuery = "SELECT 1 FORMAT JSON"
+)
+
+// HealthHandler checks the health of the Configured Upstream Origin
+func (c *Client) HealthHandler(w http.ResponseWriter, r *http.Request) {
+
+ if c.healthURL == nil {
+ c.populateHeathCheckRequestValues()
+ }
+
+ if c.healthMethod == "-" {
+ w.WriteHeader(400)
+ w.Write([]byte("Health Check URL not Configured for origin: " + c.config.Name))
+ return
+ }
+
+ req, _ := http.NewRequest(c.healthMethod, c.healthURL.String(), nil)
+ req = req.WithContext(r.Context())
+
+ req.Header = c.healthHeaders
+ engines.DoProxy(w, req)
+
+}
+
+func (c *Client) populateHeathCheckRequestValues() {
+
+ oc := c.config
+
+ if oc.HealthCheckUpstreamPath == "-" {
+ oc.HealthCheckUpstreamPath = "/"
+ }
+ if oc.HealthCheckVerb == "-" {
+ oc.HealthCheckVerb = http.MethodGet
+ }
+ if oc.HealthCheckQuery == "-" {
+ q := url.Values{"query": {healthQuery}}
+ oc.HealthCheckQuery = q.Encode()
+ }
+
+ c.healthURL = c.BaseURL()
+ c.healthURL.Path += oc.HealthCheckUpstreamPath
+ c.healthURL.RawQuery = oc.HealthCheckQuery
+ c.healthMethod = oc.HealthCheckVerb
+
+ if oc.HealthCheckHeaders != nil {
+ c.healthHeaders = http.Header{}
+ headers.UpdateHeaders(c.healthHeaders, oc.HealthCheckHeaders)
+ }
+}
diff --git a/internal/proxy/origins/clickhouse/handler_health_test.go b/internal/proxy/origins/clickhouse/handler_health_test.go
new file mode 100644
index 000000000..979c7561e
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/handler_health_test.go
@@ -0,0 +1,104 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "io/ioutil"
+ "net/http/httptest"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/proxy/request"
+ "github.com/Comcast/trickster/internal/util/metrics"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+)
+
+func init() {
+ metrics.Init()
+}
+
+func TestHealthHandler(t *testing.T) {
+
+ client := &Client{name: "test"}
+ ts, w, r, hc, err := tu.NewTestInstance("", client.DefaultPathConfigs, 200, "{}", nil, "clickhouse", "/health", "debug")
+
+ rsc := request.GetResources(r)
+ client.config = rsc.OriginConfig
+ client.webClient = hc
+ client.config.HTTPClient = hc
+ defer ts.Close()
+ if err != nil {
+ t.Error(err)
+ }
+
+ client.HealthHandler(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(bodyBytes) != "{}" {
+ t.Errorf("expected '{}' got %s.", bodyBytes)
+ }
+
+ client.healthMethod = "-"
+
+ w = httptest.NewRecorder()
+ client.HealthHandler(w, r)
+ resp = w.Result()
+ if resp.StatusCode != 400 {
+ t.Errorf("Expected status: 400 got %d.", resp.StatusCode)
+ }
+
+}
+
+func TestHealthHandlerCustomPath(t *testing.T) {
+
+ client := &Client{name: "test"}
+ ts, w, r, hc, err := tu.NewTestInstance("../../../../testdata/test.custom_health.conf", client.DefaultPathConfigs, 200, "{}", nil, "clickhouse", "/health", "debug")
+ defer ts.Close()
+ if err != nil {
+ t.Error(err)
+ }
+
+ rsc := request.GetResources(r)
+ client.config = rsc.OriginConfig
+
+ client.webClient = hc
+ client.config.HTTPClient = hc
+
+ client.HealthHandler(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(bodyBytes) != "{}" {
+ t.Errorf("expected '{}' got %s.", bodyBytes)
+ }
+
+}
diff --git a/internal/proxy/origins/clickhouse/handler_proxy.go b/internal/proxy/origins/clickhouse/handler_proxy.go
new file mode 100644
index 000000000..c3270f2c6
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/handler_proxy.go
@@ -0,0 +1,26 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+
+ "github.com/Comcast/trickster/internal/proxy/engines"
+)
+
+// ProxyHandler sends a request through the basic reverse proxy to the origin, and services non-cacheable InfluxDB API calls
+func (c *Client) ProxyHandler(w http.ResponseWriter, r *http.Request) {
+ r.URL = c.BuildUpstreamURL(r)
+ engines.DoProxy(w, r)
+}
diff --git a/internal/proxy/origins/clickhouse/handler_proxy_test.go b/internal/proxy/origins/clickhouse/handler_proxy_test.go
new file mode 100644
index 000000000..b4f6aeba8
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/handler_proxy_test.go
@@ -0,0 +1,55 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "io/ioutil"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/proxy/request"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+)
+
+func TestProxyHandler(t *testing.T) {
+
+ client := &Client{name: "test"}
+ ts, w, r, hc, err := tu.NewTestInstance("", client.DefaultPathConfigs, 200, "test", nil, "clickhouse", "/health", "debug")
+
+ rsc := request.GetResources(r)
+ client.config = rsc.OriginConfig
+ client.webClient = hc
+ client.config.HTTPClient = hc
+ defer ts.Close()
+ if err != nil {
+ t.Error(err)
+ }
+
+ client.ProxyHandler(w, r)
+ resp := w.Result()
+
+ // it should return 200 OK
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(bodyBytes) != "test" {
+ t.Errorf("expected 'test' got %s.", bodyBytes)
+ }
+
+}
diff --git a/internal/proxy/origins/clickhouse/handler_query.go b/internal/proxy/origins/clickhouse/handler_query.go
new file mode 100644
index 000000000..b2cb31ac2
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/handler_query.go
@@ -0,0 +1,36 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+ "strings"
+
+ "github.com/Comcast/trickster/internal/proxy/engines"
+)
+
+// QueryHandler handles timeseries requests for ClickHouse and processes them through the delta proxy cache
+func (c *Client) QueryHandler(w http.ResponseWriter, r *http.Request) {
+
+ rqlc := strings.Replace(strings.ToLower(r.URL.RawQuery), "%20", "+", -1)
+ // if it's not a select statement, just proxy it instead
+ if (!strings.HasPrefix(rqlc, "query=select+")) && (!(strings.Index(rqlc, "&query=select+") > 0)) &&
+ (!strings.HasSuffix(rqlc, "format+json")) {
+ c.ProxyHandler(w, r)
+ return
+ }
+
+ r.URL = c.BuildUpstreamURL(r)
+ engines.DeltaProxyCacheRequest(w, r)
+}
diff --git a/internal/proxy/origins/clickhouse/handler_query_test.go b/internal/proxy/origins/clickhouse/handler_query_test.go
new file mode 100644
index 000000000..3a40c077f
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/handler_query_test.go
@@ -0,0 +1,102 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "io/ioutil"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
+ "testing"
+
+ "github.com/Comcast/trickster/internal/proxy/request"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+)
+
+func testRawQuery() string {
+ return url.Values(map[string][]string{"query": {
+ `SELECT (intDiv(toUInt32(time_column), 60) * 60) * 1000 AS t, countMerge(some_count) AS cnt, field1, field2 ` +
+ `FROM testdb.test_table WHERE time_column BETWEEN toDateTime(1516665600) AND toDateTime(1516687200) ` +
+ `AND date_column >= toDate(1516665600) AND toDate(1516687200) ` +
+ `AND field1 > 0 AND field2 = 'some_value' GROUP BY t, field1, field2 ORDER BY t, field1 FORMAT JSON`}}).Encode()
+}
+
+func testNonSelectQuery() string {
+ return url.Values(map[string][]string{"query": {
+ `UPDATE (intDiv(toUInt32(time_column), 60) * 60) * 1000 AS t`}}).Encode()
+ // not a real query, just something to trigger a non-select proxy-only request
+}
+
+func TestQueryHandler(t *testing.T) {
+
+ client := &Client{name: "test"}
+ ts, w, r, hc, err := tu.NewTestInstance("", client.DefaultPathConfigs, 200, "{}", nil, "clickhouse", "/?"+testRawQuery(), "debug")
+ ctx := r.Context()
+ rsc := request.GetResources(r)
+ rsc.OriginClient = client
+ client.config = rsc.OriginConfig
+ client.webClient = hc
+ client.config.HTTPClient = hc
+ defer ts.Close()
+ if err != nil {
+ t.Error(err)
+ }
+
+ _, ok := client.config.Paths["/"]
+ if !ok {
+ t.Errorf("could not find path config named %s", "/")
+ }
+
+ client.QueryHandler(w, r)
+
+ resp := w.Result()
+
+ // it should return 200 OK
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(bodyBytes) != "{}" {
+ t.Errorf("expected '{}' got %s.", bodyBytes)
+ }
+
+ r, _ = http.NewRequest(http.MethodGet, ts.URL+"/?"+testNonSelectQuery(), nil)
+ w = httptest.NewRecorder()
+
+ r = r.WithContext(ctx)
+
+ client.QueryHandler(w, r)
+
+ resp = w.Result()
+
+ // it should return 200 OK
+ if resp.StatusCode != 200 {
+ t.Errorf("expected 200 got %d.", resp.StatusCode)
+ }
+
+ bodyBytes, err = ioutil.ReadAll(resp.Body)
+ if err != nil {
+ t.Error(err)
+ }
+
+ if string(bodyBytes) != "{}" {
+ t.Errorf("expected '{}' got %s.", bodyBytes)
+ }
+
+}
diff --git a/internal/proxy/origins/clickhouse/model.go b/internal/proxy/origins/clickhouse/model.go
new file mode 100644
index 000000000..f3fb69c70
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/model.go
@@ -0,0 +1,337 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "sort"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/pkg/sort/times"
+)
+
+const (
+ millisPerSecond = int64(time.Second / time.Millisecond)
+ nanosPerMillisecond = int64(time.Millisecond / time.Nanosecond)
+)
+
+func msToTime(ms string) (time.Time, error) {
+ msInt, err := strconv.ParseInt(ms, 10, 64)
+ if err != nil {
+ return time.Time{}, err
+ }
+ return time.Unix(msInt/millisPerSecond,
+ (msInt%millisPerSecond)*nanosPerMillisecond), nil
+}
+
+// FieldDefinition ...
+type FieldDefinition struct {
+ Name string `json:"name"`
+ Type string `json:"type"`
+}
+
+// ResponseValue ...
+type ResponseValue map[string]interface{}
+
+// DataSet ...
+type DataSet struct {
+ Metric map[string]interface{}
+ Points []Point
+}
+
+// Points ...
+type Points []Point
+
+// Point ...
+type Point struct {
+ Timestamp time.Time
+ Value float64
+}
+
+// Response is the JSON responose document structure for ClickHouse query results
+type Response struct {
+ Meta []FieldDefinition `json:"meta"`
+ RawData []ResponseValue `json:"data"`
+ Rows int `json:"rows"`
+ Order []string `json:"-"`
+ StepDuration time.Duration `json:"step,omitempty"`
+ ExtentList timeseries.ExtentList `json:"extents,omitempty"`
+}
+
+// ResultsEnvelope is the ClickHouse document structure optimized for time series manipulation
+type ResultsEnvelope struct {
+ Meta []FieldDefinition `json:"meta"`
+ Data map[string]*DataSet `json:"data"`
+ StepDuration time.Duration `json:"step,omitempty"`
+ ExtentList timeseries.ExtentList `json:"extents,omitempty"`
+ Serializers map[string]func(interface{}) `json:"-"`
+ SeriesOrder []string `json:"series_order,omitempty"`
+
+ timestamps map[time.Time]bool // tracks unique timestamps in the matrix data
+ tslist times.Times
+ isSorted bool // tracks if the matrix data is currently sorted
+ isCounted bool // tracks if timestamps slice is up-to-date
+}
+
+// MarshalTimeseries converts a Timeseries into a JSON blob
+func (c *Client) MarshalTimeseries(ts timeseries.Timeseries) ([]byte, error) {
+ // Marshal the Envelope back to a json object for Cache Storage
+ return json.Marshal(ts.(*ResultsEnvelope))
+}
+
+// UnmarshalTimeseries converts a JSON blob into a Timeseries
+func (c *Client) UnmarshalTimeseries(data []byte) (timeseries.Timeseries, error) {
+ re := &ResultsEnvelope{}
+ err := json.Unmarshal(data, re)
+ return re, err
+}
+
+// Parts ...
+func (rv ResponseValue) Parts(timeKey, valKey string) (string, time.Time, float64, ResponseValue) {
+
+ if len(rv) < 3 {
+ return noParts()
+ }
+
+ labels := make([]string, 0, len(rv)-2)
+ var t time.Time
+ var val float64
+ var err error
+
+ meta := make(ResponseValue)
+
+ for k, v := range rv {
+ switch k {
+ case timeKey:
+ t, err = msToTime(v.(string))
+ if err != nil {
+ return noParts()
+ }
+ case valKey:
+ if av, ok := v.(float64); ok {
+ val = av
+ continue
+ }
+ val, err = strconv.ParseFloat(v.(string), 64)
+ if err != nil {
+ return noParts()
+ }
+ default:
+ meta[k] = v
+ labels = append(labels, fmt.Sprintf("%s=%v", k, v))
+ }
+ }
+ sort.Strings(labels)
+ return fmt.Sprintf("{%s}", strings.Join(labels, ";")), t, val, meta
+}
+
+func noParts() (string, time.Time, float64, ResponseValue) {
+ return "{}", time.Time{}, 0.0, ResponseValue{}
+}
+
+// MarshalJSON ...
+func (re ResultsEnvelope) MarshalJSON() ([]byte, error) {
+
+ if len(re.Meta) < 2 {
+ return nil, fmt.Errorf("Must have at least two fields; only have %d", len(re.Meta))
+ }
+
+ var mpl, fl int
+ for _, v := range re.Data {
+ lp := len(v.Points)
+ fl += lp
+ if mpl < lp {
+ mpl = lp
+ }
+ }
+
+ rsp := &Response{
+ Meta: re.Meta,
+ RawData: make([]ResponseValue, 0, fl),
+ Rows: re.ValueCount(),
+ StepDuration: re.StepDuration,
+ ExtentList: re.ExtentList,
+ }
+
+ rsp.Order = make([]string, 0, len(re.Meta))
+ for _, k := range re.Meta {
+ rsp.Order = append(rsp.Order, k.Name)
+ }
+
+ // Assume the first item in the meta array is the time, and the second is the value
+ timestampFieldName := rsp.Order[0]
+ valueFieldName := rsp.Order[1]
+
+ tm := make(map[time.Time][]ResponseValue)
+ tl := make(times.Times, 0, mpl)
+
+ l := len(re.Data)
+
+ prepareMarshalledPoints := func(ds *DataSet) {
+
+ var ok bool
+ var t []ResponseValue
+
+ for _, p := range ds.Points {
+
+ t, ok = tm[p.Timestamp]
+ if !ok {
+ tl = append(tl, p.Timestamp)
+ t = make([]ResponseValue, 0, l)
+ }
+
+ r := ResponseValue{
+ timestampFieldName: strconv.FormatInt(p.Timestamp.UnixNano()/int64(time.Millisecond), 10),
+ valueFieldName: strconv.FormatFloat(p.Value, 'f', -1, 64),
+ }
+ for k2, v2 := range ds.Metric {
+ r[k2] = v2
+ }
+
+ t = append(t, r)
+ tm[p.Timestamp] = t
+ }
+ }
+
+ for _, key := range re.SeriesOrder {
+ if ds, ok := re.Data[key]; ok {
+ prepareMarshalledPoints(ds)
+ }
+ }
+
+ sort.Sort(tl)
+
+ for _, t := range tl {
+ rsp.RawData = append(rsp.RawData, tm[t]...)
+ }
+
+ bytes, err := json.Marshal(rsp)
+ if err != nil {
+ return nil, err
+ }
+
+ return bytes, nil
+}
+
+// MarshalJSON ...
+func (rsp *Response) MarshalJSON() ([]byte, error) {
+
+ buf := &bytes.Buffer{}
+ buf.WriteString(`{"meta":`)
+ meta, _ := json.Marshal(rsp.Meta)
+ buf.Write(meta)
+ buf.WriteString(`,"data":[`)
+ d := make([]string, 0, len(rsp.RawData))
+ for _, rd := range rsp.RawData {
+ d = append(d, string(rd.ToJSON(rsp.Order)))
+ }
+ buf.WriteString(strings.Join(d, ",") + "]")
+ buf.WriteString(fmt.Sprintf(`,"rows": %d`, rsp.Rows))
+
+ if rsp.ExtentList != nil && len(rsp.ExtentList) > 0 {
+ el, _ := json.Marshal(rsp.ExtentList)
+ buf.WriteString(fmt.Sprintf(`,"extents": %s`, string(el)))
+ }
+
+ buf.WriteString("}")
+
+ b := buf.Bytes()
+
+ return b, nil
+}
+
+// ToJSON ...
+func (rv ResponseValue) ToJSON(order []string) []byte {
+ buf := &bytes.Buffer{}
+ buf.WriteString("{")
+ lines := make([]string, 0, len(rv))
+ for _, k := range order {
+ if v, ok := rv[k]; ok {
+
+ // cleanup here
+ j, err := json.Marshal(v)
+ if err != nil {
+ continue
+ }
+ lines = append(lines, fmt.Sprintf(`"%s":%s`, k, string(j)))
+ }
+ }
+ buf.WriteString(strings.Join(lines, ",") + "}")
+ return buf.Bytes()
+}
+
+// UnmarshalJSON ...
+func (re *ResultsEnvelope) UnmarshalJSON(b []byte) error {
+
+ response := Response{}
+ err := json.Unmarshal(b, &response)
+ if err != nil {
+ return err
+ }
+
+ if len(response.Meta) < 2 {
+ return fmt.Errorf("Must have at least two fields; only have %d", len(response.Meta))
+ }
+
+ re.Meta = response.Meta
+ re.ExtentList = response.ExtentList
+ re.StepDuration = response.StepDuration
+ re.SeriesOrder = make([]string, 0)
+
+ // Assume the first item in the meta array is the time field, and the second is the value field
+ timestampFieldName := response.Meta[0].Name
+ valueFieldName := response.Meta[1].Name
+
+ registeredMetrics := make(map[string]bool)
+
+ re.Data = make(map[string]*DataSet)
+ l := len(response.RawData)
+ for _, v := range response.RawData {
+ metric, ts, val, meta := v.Parts(timestampFieldName, valueFieldName)
+ if _, ok := registeredMetrics[metric]; !ok {
+ registeredMetrics[metric] = true
+ re.SeriesOrder = append(re.SeriesOrder, metric)
+ }
+ if !ts.IsZero() {
+ a, ok := re.Data[metric]
+ if !ok {
+ a = &DataSet{Metric: meta, Points: make([]Point, 0, l)}
+ }
+ a.Points = append(a.Points, Point{Timestamp: ts, Value: val})
+ re.Data[metric] = a
+ }
+ }
+
+ return nil
+}
+
+// Len returns the length of a slice of time series data points
+func (p Points) Len() int {
+ return len(p)
+}
+
+// Less returns true if i comes before j
+func (p Points) Less(i, j int) bool {
+ return p[i].Timestamp.Before(p[j].Timestamp)
+}
+
+// Swap modifies a slice of time series data points by swapping the values in indexes i and j
+func (p Points) Swap(i, j int) {
+ p[i], p[j] = p[j], p[i]
+}
diff --git a/internal/proxy/origins/clickhouse/model_test.go b/internal/proxy/origins/clickhouse/model_test.go
new file mode 100644
index 000000000..f94baa10b
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/model_test.go
@@ -0,0 +1,333 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "encoding/json"
+ "reflect"
+ "sort"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/timeseries"
+)
+
+func TestParts(t *testing.T) {
+
+ rv1 := ResponseValue{
+ "t": "1557766080000",
+ "cnt": "27",
+ "meta1": 200,
+ "meta2": "value3",
+ }
+
+ metric, ts, val, _ := rv1.Parts("t", "cnt")
+
+ expectedTs := time.Unix(1557766080, 0)
+ expectedMetric := "{meta1=200;meta2=value3}"
+ var expectedValue float64 = 27
+
+ if ts != expectedTs {
+ t.Errorf("expected %d got %d", expectedTs.Unix(), ts.Unix())
+ }
+
+ if metric != expectedMetric {
+ t.Errorf("expected %s got %s", expectedMetric, metric)
+ }
+
+ if val != expectedValue {
+ t.Errorf("expected %f got %f", expectedValue, val)
+ }
+
+ rv2 := ResponseValue{
+ "t": "1557766080000",
+ "cnt": "27",
+ }
+
+ metric, _, _, _ = rv2.Parts("t", "cnt")
+ if metric != "{}" {
+ t.Errorf("expected '{}' got %s", metric)
+ }
+
+ rv3 := ResponseValue{
+ "t": "A557766080000",
+ "cnt": "27",
+ "meta1": 200,
+ }
+
+ metric, _, _, _ = rv3.Parts("t", "cnt")
+ if metric != "{}" {
+ t.Errorf("expected '{}' got %s", metric)
+ }
+
+ rv4 := ResponseValue{
+ "t": "1557766080000",
+ "cnt": "2a7",
+ "meta1": 200,
+ }
+
+ metric, _, _, _ = rv4.Parts("t", "cnt")
+ if metric != "{}" {
+ t.Errorf("expected '{}' got %s", metric)
+ }
+
+ rv5 := ResponseValue{
+ "t": "1557766080000",
+ "cnt": 27.5,
+ "meta1": 200,
+ }
+
+ metric, _, _, _ = rv5.Parts("t", "cnt")
+ if metric != "{meta1=200}" {
+ t.Errorf("expected '{meta1=200}' got %s", metric)
+ }
+
+}
+
+var testJSON1 = []byte(`{"meta":[{"name":"t","type":"UInt64"},{"name":"cnt","type":"UInt64"},{"name":"meta1","type":"UInt16"},{"name":"meta2","type":"String"}],"data":[{"t":"1557766080000","cnt":"12648509","meta1":200,"meta2":"value2"},{"t":"1557766080000","cnt":"10260032","meta1":200,"meta2":"value3"},{"t":"1557766080000","cnt":"1","meta1":206,"meta2":"value3"}],"rows":3}`)
+var testJSON2 = []byte(`{"meta":[{"name":"t"}],"data":[{"t":"1557766080000","cnt":"12648509","meta1":200,"meta2":"value2"},{"t":"1557766080000","cnt":"10260032","meta1":200,"meta2":"value3"},{"t":"1557766080000","cnt":"1","meta1":206,"meta2":"value3"}],"rows":3}`) // should generate error
+
+var testRE1 = &ResultsEnvelope{
+ Meta: []FieldDefinition{
+ {
+ Name: "t",
+ Type: "UInt64",
+ },
+ {
+ Name: "cnt",
+ Type: "UInt64",
+ },
+ {
+ Name: "meta1",
+ Type: "UInt16",
+ },
+ {
+ Name: "meta2",
+ Type: "String",
+ },
+ },
+
+ SeriesOrder: []string{"1", "2", "3"},
+
+ Data: map[string]*DataSet{
+ "1": {
+ Metric: map[string]interface{}{
+ "meta1": 200,
+ "meta2": "value2",
+ },
+ Points: []Point{
+ {
+ Timestamp: time.Unix(1557766080, 0),
+ Value: 12648509,
+ },
+ },
+ },
+ "2": {
+ Metric: map[string]interface{}{
+ "meta1": 200,
+ "meta2": "value3",
+ },
+ Points: []Point{
+ {
+ Timestamp: time.Unix(1557766080, 0),
+ Value: 10260032,
+ },
+ },
+ },
+ "3": {
+ Metric: map[string]interface{}{
+ "meta1": 206,
+ "meta2": "value3",
+ },
+ Points: []Point{
+ {
+ Timestamp: time.Unix(1557766080, 0),
+ Value: 1,
+ },
+ },
+ },
+ },
+}
+
+func TestREMarshalJSON(t *testing.T) {
+
+ expectedLen := len(testJSON1)
+
+ re := ResultsEnvelope{}
+ err := re.UnmarshalJSON(testJSON1)
+ if err != nil {
+ t.Error(err)
+ }
+
+ bytes, err := re.MarshalJSON()
+ if err != nil {
+ t.Error(err)
+ }
+
+ if len(bytes) != expectedLen {
+ t.Errorf("expected %d got %d", expectedLen, len(bytes))
+ }
+
+ re.Meta = re.Meta[:0]
+ _, err = re.MarshalJSON()
+ if err == nil {
+ t.Errorf("expected error: %s", `Must have at least two fields; only have 0`)
+ }
+
+}
+
+func TestRSPMarshalJSON(t *testing.T) {
+
+ rsp := &Response{ExtentList: timeseries.ExtentList{{Start: time.Unix(0, 0), End: time.Unix(5, 0)}}}
+
+ bytes, err := rsp.MarshalJSON()
+ if err != nil {
+ t.Error(err)
+ return
+ }
+
+ rsp1 := &Response{}
+ json.Unmarshal(bytes, rsp1)
+
+ if rsp.ExtentList[0].Start.Unix() != rsp1.ExtentList[0].Start.Unix() {
+ t.Errorf("expected %d got %d", rsp.ExtentList[0].Start.Unix(), rsp.ExtentList[0].Start.Unix())
+ }
+
+ if rsp.ExtentList[0].End.Unix() != rsp1.ExtentList[0].End.Unix() {
+ t.Errorf("expected %d got %d", rsp.ExtentList[0].End.Unix(), rsp.ExtentList[0].End.Unix())
+ }
+
+}
+
+func TestUnmarshalTimeseries(t *testing.T) {
+
+ client := &Client{}
+ ts, err := client.UnmarshalTimeseries(testJSON1)
+ if err != nil {
+ t.Error(err)
+ return
+ }
+
+ re := ts.(*ResultsEnvelope)
+
+ if len(re.Meta) != 4 {
+ t.Errorf(`expected 4. got %d`, len(re.Meta))
+ return
+ }
+
+ if len(re.Data) != 3 {
+ t.Errorf(`expected 3. got %d`, len(re.Data))
+ return
+ }
+
+ _, err = client.UnmarshalTimeseries(nil)
+ if err == nil {
+ t.Errorf("expected error: %s", `unexpected end of JSON input`)
+ return
+ }
+
+ _, err = client.UnmarshalTimeseries(testJSON2)
+ if err == nil {
+ t.Errorf("expected error: %s", `Must have at least two fields; only have 1`)
+ return
+ }
+
+}
+
+func TestMarshalTimeseries(t *testing.T) {
+ expectedLen := len(testJSON1)
+ client := &Client{}
+ bytes, err := client.MarshalTimeseries(testRE1)
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ if !reflect.DeepEqual(testJSON1, bytes) {
+ t.Errorf("expected %d got %d", expectedLen, len(bytes))
+ }
+}
+
+func TestUnmarshalJSON(t *testing.T) {
+
+ re := ResultsEnvelope{}
+ err := re.UnmarshalJSON(testJSON1)
+ if err != nil {
+ t.Error(err)
+ return
+ }
+
+ if len(re.Meta) != 4 {
+ t.Errorf(`expected 4. got %d`, len(re.Meta))
+ return
+ }
+
+ m := re.Meta[2]
+ if m.Name != "meta1" {
+ t.Errorf(`expected meta1 found %s`, m.Name)
+ return
+ }
+
+ if len(re.Data) != 3 {
+ t.Errorf(`expected 3. got %d`, len(re.Data))
+ return
+ }
+
+ key := "{meta1=206;meta2=value3}"
+ v, ok := re.Data[key]
+ if !ok {
+ t.Errorf(`expected to find key %s`, key)
+ return
+ }
+
+ if len(v.Points) != 1 {
+ t.Errorf(`expected 1 got %d`, len(v.Points))
+ return
+ }
+
+ if v.Points[0].Value != 1 {
+ t.Errorf(`expected 1 got %f`, v.Points[0].Value)
+ return
+ }
+
+ err = re.UnmarshalJSON(nil)
+ if err == nil {
+ t.Errorf("expected error: %s", `unexpected end of JSON input`)
+ return
+ }
+
+ err = re.UnmarshalJSON(testJSON2)
+ if err == nil {
+ t.Errorf("expected error: %s", `Must have at least two fields; only have 1`)
+ return
+ }
+
+}
+
+func TestMSToTime(t *testing.T) {
+ _, err := msToTime("bad")
+ if err == nil {
+ t.Errorf("expected error for invalid syntax")
+ }
+}
+
+func TestSortPoints(t *testing.T) {
+
+ p := Points{{Timestamp: time.Unix(1, 0), Value: 12}, {Timestamp: time.Unix(0, 0), Value: 13}, {Timestamp: time.Unix(2, 0), Value: 22}}
+ sort.Sort(p)
+
+ if p[0].Timestamp.Unix() != 0 {
+ t.Errorf("expected %d got %d", 0, p[0].Timestamp.Unix())
+ }
+
+}
diff --git a/internal/proxy/origins/clickhouse/routes.go b/internal/proxy/origins/clickhouse/routes.go
new file mode 100644
index 000000000..a3d99545f
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/routes.go
@@ -0,0 +1,54 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+
+ "github.com/Comcast/trickster/internal/config"
+)
+
+func (c *Client) registerHandlers() {
+ c.handlersRegistered = true
+ c.handlers = make(map[string]http.Handler)
+ // This is the registry of handlers that Trickster supports for InfluxDB,
+ // and are able to be referenced by name (map key) in Config Files
+ c.handlers["health"] = http.HandlerFunc(c.HealthHandler)
+ c.handlers["query"] = http.HandlerFunc(c.QueryHandler)
+ c.handlers["proxy"] = http.HandlerFunc(c.ProxyHandler)
+}
+
+// Handlers returns a map of the HTTP Handlers the client has registered
+func (c *Client) Handlers() map[string]http.Handler {
+ if !c.handlersRegistered {
+ c.registerHandlers()
+ }
+ return c.handlers
+}
+
+// DefaultPathConfigs returns the default PathConfigs for the given OriginType
+func (c *Client) DefaultPathConfigs(oc *config.OriginConfig) map[string]*config.PathConfig {
+ paths := map[string]*config.PathConfig{
+ "/": {
+ Path: "/",
+ HandlerName: "query",
+ Methods: []string{http.MethodGet, http.MethodPost},
+ MatchType: config.PathMatchTypePrefix,
+ MatchTypeName: "prefix",
+ CacheKeyParams: []string{"query", "database"},
+ OriginConfig: oc,
+ },
+ }
+ return paths
+}
diff --git a/internal/proxy/origins/clickhouse/routes_test.go b/internal/proxy/origins/clickhouse/routes_test.go
new file mode 100644
index 000000000..85b265b54
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/routes_test.go
@@ -0,0 +1,60 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "testing"
+
+ "github.com/Comcast/trickster/internal/proxy/request"
+ tu "github.com/Comcast/trickster/internal/util/testing"
+)
+
+func TestRegisterHandlers(t *testing.T) {
+ c := &Client{}
+ c.registerHandlers()
+ if _, ok := c.handlers["query"]; !ok {
+ t.Errorf("expected to find handler named: %s", "query")
+ }
+}
+
+func TestHandlers(t *testing.T) {
+ c := &Client{}
+ m := c.Handlers()
+ if _, ok := m["query"]; !ok {
+ t.Errorf("expected to find handler named: %s", "query")
+ }
+}
+
+func TestDefaultPathConfigs(t *testing.T) {
+
+ client := &Client{name: "test"}
+ ts, _, r, hc, err := tu.NewTestInstance("", client.DefaultPathConfigs, 204, "", nil, "clickhouse", "/", "debug")
+ rsc := request.GetResources(r)
+ client.config = rsc.OriginConfig
+ client.webClient = hc
+ defer ts.Close()
+ if err != nil {
+ t.Error(err)
+ }
+
+ if _, ok := client.config.Paths["/"]; !ok {
+ t.Errorf("expected to find path named: %s", "/")
+ }
+
+ const expectedLen = 1
+ if len(client.config.Paths) != expectedLen {
+ t.Errorf("expected %d got %d", expectedLen, len(client.config.Paths))
+ }
+
+}
diff --git a/internal/proxy/origins/clickhouse/series.go b/internal/proxy/origins/clickhouse/series.go
new file mode 100644
index 000000000..856d73ab5
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/series.go
@@ -0,0 +1,509 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "sort"
+ "sync"
+ "time"
+
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/pkg/sort/times"
+)
+
+// Step returns the step for the Timeseries
+func (re *ResultsEnvelope) Step() time.Duration {
+ return re.StepDuration
+}
+
+// SetStep sets the step for the Timeseries
+func (re *ResultsEnvelope) SetStep(step time.Duration) {
+ re.StepDuration = step
+}
+
+// Merge merges the provided Timeseries list into the base Timeseries (in the order provided) and optionally sorts the merged Timeseries
+func (re *ResultsEnvelope) Merge(sort bool, collection ...timeseries.Timeseries) {
+
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+
+ for _, ts := range collection {
+ if ts != nil {
+ re2 := ts.(*ResultsEnvelope)
+ for k, s := range re2.Data {
+ wg.Add(1)
+ go func(l string, d *DataSet) {
+ mtx.Lock()
+ if _, ok := re.Data[l]; !ok {
+ re.Data[l] = d
+ mtx.Unlock()
+ wg.Done()
+ return
+ }
+ re.Data[l].Points = append(re.Data[l].Points, d.Points...)
+ mtx.Unlock()
+ wg.Done()
+ }(k, s)
+ }
+ wg.Wait()
+ re.mergeSeriesOrder(re2.SeriesOrder)
+ re.ExtentList = append(re.ExtentList, re2.ExtentList...)
+ }
+ }
+
+ re.ExtentList = re.ExtentList.Compress(re.StepDuration)
+ re.isSorted = false
+ re.isCounted = false
+ if sort {
+ re.Sort()
+ }
+}
+
+func (re *ResultsEnvelope) mergeSeriesOrder(so2 []string) {
+
+ if len(so2) == 0 {
+ return
+ }
+
+ if len(re.SeriesOrder) == 0 {
+ re.SeriesOrder = so2
+ return
+ }
+
+ so1 := make([]string, len(re.SeriesOrder), len(re.SeriesOrder)+len(so2))
+ copy(so1, re.SeriesOrder)
+ adds := make([]string, 0, len(so2))
+ added := make(map[string]bool)
+
+ for _, n := range so2 {
+ if _, ok := re.Data[n]; !ok {
+ if _, ok2 := added[n]; !ok2 {
+ adds = append(adds, n)
+ added[n] = true
+ }
+ continue
+ }
+
+ if len(adds) > 0 {
+ for i, v := range so1 {
+ if v == n {
+ adds = append(adds, so1[i:]...)
+ so1 = append(so1[0:i], adds...)
+ }
+ }
+ adds = adds[:0]
+ }
+ }
+
+ if len(adds) > 0 {
+ so1 = append(so1, adds...)
+ }
+
+ re.SeriesOrder = so1
+
+}
+
+// Clone returns a perfect copy of the base Timeseries
+func (re *ResultsEnvelope) Clone() timeseries.Timeseries {
+ re2 := &ResultsEnvelope{
+ isCounted: re.isCounted,
+ isSorted: re.isSorted,
+ StepDuration: re.StepDuration,
+ }
+
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+
+ if re.SeriesOrder != nil {
+ re2.SeriesOrder = make([]string, len(re.SeriesOrder))
+ copy(re2.SeriesOrder, re.SeriesOrder)
+ }
+
+ if re.ExtentList != nil {
+ re2.ExtentList = make(timeseries.ExtentList, len(re.ExtentList))
+ copy(re2.ExtentList, re.ExtentList)
+ }
+
+ if re.tslist != nil {
+ re2.tslist = make(times.Times, len(re.tslist))
+ copy(re2.tslist, re.tslist)
+ }
+
+ if re.Meta != nil {
+ re2.Meta = make([]FieldDefinition, len(re.Meta))
+ copy(re2.Meta, re.Meta)
+ }
+
+ if re.Serializers != nil {
+ re2.Serializers = make(map[string]func(interface{}))
+ wg.Add(1)
+ go func() {
+ for k, s := range re.Serializers {
+ re2.Serializers[k] = s
+ }
+ wg.Done()
+ }()
+ }
+
+ if re.timestamps != nil {
+ re2.timestamps = make(map[time.Time]bool)
+ for k, v := range re.timestamps {
+ wg.Add(1)
+ go func(t time.Time, b bool) {
+ mtx.Lock()
+ re2.timestamps[t] = b
+ mtx.Unlock()
+ wg.Done()
+ }(k, v)
+ }
+ }
+
+ if re.Data != nil {
+ re2.Data = make(map[string]*DataSet)
+ wg.Add(1)
+ go func() {
+ for k, ds := range re.Data {
+ ds2 := &DataSet{Metric: make(map[string]interface{})}
+ for l, v := range ds.Metric {
+ ds2.Metric[l] = v
+ }
+ ds2.Points = ds.Points[:]
+ re2.Data[k] = ds2
+ }
+ wg.Done()
+ }()
+ }
+
+ wg.Wait()
+
+ return re2
+}
+
+// CropToSize reduces the number of elements in the Timeseries to the provided count, by evicting elements
+// using a least-recently-used methodology. Any timestamps newer than the provided time are removed before
+// sizing, in order to support backfill tolerance. The provided extent will be marked as used during crop.
+func (re *ResultsEnvelope) CropToSize(sz int, t time.Time, lur timeseries.Extent) {
+ re.isCounted = false
+ re.isSorted = false
+ x := len(re.ExtentList)
+ // The Series has no extents, so no need to do anything
+ if x < 1 {
+ re.Data = make(map[string]*DataSet)
+ re.ExtentList = timeseries.ExtentList{}
+ return
+ }
+
+ // Crop to the Backfill Tolerance Value if needed
+ if re.ExtentList[x-1].End.After(t) {
+ re.CropToRange(timeseries.Extent{Start: re.ExtentList[0].Start, End: t})
+ }
+
+ tc := re.TimestampCount()
+ el := timeseries.ExtentListLRU(re.ExtentList).UpdateLastUsed(lur, re.StepDuration)
+ sort.Sort(el)
+ if len(re.Data) == 0 || tc <= sz {
+ return
+ }
+
+ rc := tc - sz // # of required timestamps we must delete to meet the rentention policy
+ removals := make(map[time.Time]bool)
+ done := false
+ var ok bool
+
+ for _, x := range el {
+ for ts := x.Start; !x.End.Before(ts) && !done; ts = ts.Add(re.StepDuration) {
+ if _, ok = re.timestamps[ts]; ok {
+ removals[ts] = true
+ done = len(removals) >= rc
+ }
+ }
+ if done {
+ break
+ }
+ }
+
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+
+ for _, s := range re.Data {
+ tmp := s.Points[:0]
+ for _, r := range s.Points {
+ wg.Add(1)
+ go func(p Point) {
+ mtx.Lock()
+ if _, ok := removals[p.Timestamp]; !ok {
+ tmp = append(tmp, p)
+ }
+ mtx.Unlock()
+ wg.Done()
+ }(r)
+ }
+ wg.Wait()
+ s.Points = tmp
+ }
+
+ tl := times.FromMap(removals)
+ sort.Sort(tl)
+
+ for _, t := range tl {
+ for i, e := range el {
+ if e.StartsAt(t) {
+ el[i].Start = e.Start.Add(re.StepDuration)
+ }
+ }
+ }
+ wg.Wait()
+
+ re.ExtentList = timeseries.ExtentList(el).Compress(re.StepDuration)
+ re.Sort()
+}
+
+// CropToRange reduces the Timeseries down to timestamps contained within the provided Extents (inclusive).
+// CropToRange assumes the base Timeseries is already sorted, and will corrupt an unsorted Timeseries
+func (re *ResultsEnvelope) CropToRange(e timeseries.Extent) {
+ re.isCounted = false
+ x := len(re.ExtentList)
+ // The Series has no extents, so no need to do anything
+ if x < 1 {
+ re.Data = make(map[string]*DataSet)
+ re.ExtentList = timeseries.ExtentList{}
+ return
+ }
+
+ // if the extent of the series is entirely outside the extent of the crop range, return empty set and bail
+ if re.ExtentList.OutsideOf(e) {
+ re.Data = make(map[string]*DataSet)
+ re.ExtentList = timeseries.ExtentList{}
+ return
+ }
+
+ // if the series extent is entirely inside the extent of the crop range, simply adjust down its ExtentList
+ if re.ExtentList.InsideOf(e) {
+ if re.ValueCount() == 0 {
+ re.Data = make(map[string]*DataSet)
+ }
+ re.ExtentList = re.ExtentList.Crop(e)
+ return
+ }
+
+ if len(re.Data) == 0 {
+ re.ExtentList = re.ExtentList.Crop(e)
+ return
+ }
+
+ deletes := make(map[string]bool)
+
+ for i, s := range re.Data {
+ start := -1
+ end := -1
+ for j, val := range s.Points {
+ t := val.Timestamp
+ if t.Equal(e.End) {
+ // for cases where the first element is the only qualifying element,
+ // start must be incremented or an empty response is returned
+ if j == 0 || t.Equal(e.Start) || start == -1 {
+ start = j
+ }
+ end = j + 1
+ break
+ }
+ if t.After(e.End) {
+ end = j
+ break
+ }
+ if t.Before(e.Start) {
+ continue
+ }
+ if start == -1 && (t.Equal(e.Start) || (e.End.After(t) && t.After(e.Start))) {
+ start = j
+ }
+ }
+ if start != -1 && len(s.Points) > 0 {
+ if end == -1 {
+ end = len(s.Points)
+ }
+ re.Data[i].Points = s.Points[start:end]
+ } else {
+ deletes[i] = true
+ }
+ }
+
+ for i := range deletes {
+ delete(re.Data, i)
+ }
+
+ re.ExtentList = re.ExtentList.Crop(e)
+}
+
+// Sort sorts all Values in each Series chronologically by their timestamp
+func (re *ResultsEnvelope) Sort() {
+
+ if re.isSorted || len(re.Data) == 0 {
+ return
+ }
+
+ tsm := map[time.Time]bool{}
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+
+ for i, s := range re.Data {
+ m := make(map[time.Time]Point)
+ keys := make(times.Times, 0, len(s.Points))
+ for _, v := range s.Points {
+ wg.Add(1)
+ go func(sp Point) {
+ mtx.Lock()
+ if _, ok := m[sp.Timestamp]; !ok {
+ keys = append(keys, sp.Timestamp)
+ m[sp.Timestamp] = sp
+ }
+ tsm[sp.Timestamp] = true
+ mtx.Unlock()
+ wg.Done()
+ }(v)
+ }
+ wg.Wait()
+ sort.Sort(keys)
+ sm := make(Points, 0, len(keys))
+ for _, key := range keys {
+ sm = append(sm, m[key])
+ }
+ re.Data[i].Points = sm
+ }
+
+ sort.Sort(re.ExtentList)
+
+ re.timestamps = tsm
+ re.tslist = times.FromMap(tsm)
+ re.isCounted = true
+ re.isSorted = true
+}
+
+func (re *ResultsEnvelope) updateTimestamps() {
+
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+
+ if re.isCounted {
+ return
+ }
+ m := make(map[time.Time]bool)
+ for _, s := range re.Data {
+ for _, v := range s.Points {
+ wg.Add(1)
+ go func(t time.Time) {
+ mtx.Lock()
+ m[t] = true
+ mtx.Unlock()
+ wg.Done()
+ }(v.Timestamp)
+ }
+ }
+ wg.Wait()
+ re.timestamps = m
+ re.tslist = times.FromMap(m)
+ re.isCounted = true
+}
+
+// SetExtents overwrites a Timeseries's known extents with the provided extent list
+func (re *ResultsEnvelope) SetExtents(extents timeseries.ExtentList) {
+ re.isCounted = false
+ re.ExtentList = extents
+}
+
+// Extents returns the Timeseries's ExentList
+func (re *ResultsEnvelope) Extents() timeseries.ExtentList {
+ return re.ExtentList
+}
+
+// TimestampCount returns the number of unique timestamps across the timeseries
+func (re *ResultsEnvelope) TimestampCount() int {
+ re.updateTimestamps()
+ return len(re.timestamps)
+}
+
+// SeriesCount returns the number of individual Series in the Timeseries object
+func (re *ResultsEnvelope) SeriesCount() int {
+ return len(re.Data)
+}
+
+// ValueCount returns the count of all values across all Series in the Timeseries object
+func (re *ResultsEnvelope) ValueCount() int {
+ c := 0
+ wg := sync.WaitGroup{}
+ mtx := sync.Mutex{}
+ for i := range re.Data {
+ wg.Add(1)
+ go func(j int) {
+ mtx.Lock()
+ c += j
+ mtx.Unlock()
+ wg.Done()
+ }(len(re.Data[i].Points))
+ }
+ wg.Wait()
+ return c
+}
+
+// Size returns the approximate memory utilization in bytes of the timeseries
+func (re *ResultsEnvelope) Size() int {
+
+ var size int
+ wg := sync.WaitGroup{}
+
+ var a int
+ ma := sync.Mutex{}
+ for i := range re.Meta {
+ wg.Add(1)
+ go func(j int) {
+ ma.Lock()
+ a += len(re.Meta[j].Name) + len(re.Meta[j].Type)
+ ma.Unlock()
+ wg.Done()
+ }(i)
+ }
+
+ var b int
+ mb := sync.Mutex{}
+ for k, v := range re.Data {
+ b += len(k)
+ wg.Add(1)
+ go func(d *DataSet) {
+ mb.Lock()
+ b += len(d.Points) * 16
+ mb.Unlock()
+ wg.Done()
+ }(v)
+ }
+
+ var c int
+ mc := sync.Mutex{}
+ for _, s := range re.SeriesOrder {
+ wg.Add(1)
+ go func(t string) {
+ mc.Lock()
+ c += len(t)
+ mc.Unlock()
+ wg.Done()
+ }(s)
+ }
+
+ // ExtentList + StepDuration + Timestamps + Times + isCounted + isSorted
+ d := (len(re.ExtentList) * 24) + 8 + (len(re.timestamps) * 9) + (len(re.tslist) * 8) + 2
+
+ wg.Wait()
+ size = a + b + c + d
+ return size
+
+}
diff --git a/internal/proxy/origins/clickhouse/series_test.go b/internal/proxy/origins/clickhouse/series_test.go
new file mode 100644
index 000000000..3fe9c9fbe
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/series_test.go
@@ -0,0 +1,1505 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "reflect"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/pkg/sort/times"
+)
+
+func TestSetStep(t *testing.T) {
+ re := ResultsEnvelope{}
+ const step = time.Duration(300) * time.Minute
+ re.SetStep(step)
+ if re.StepDuration != step {
+ t.Errorf(`expected "%s". got "%s"`, testStep, re.StepDuration)
+ }
+}
+
+func TestStep(t *testing.T) {
+ re := ResultsEnvelope{}
+ const step = time.Duration(300) * time.Minute
+ re.SetStep(step)
+ if re.Step() != step {
+ t.Errorf(`expected "%s". got "%s"`, testStep, re.Step())
+ }
+}
+
+func TestMerge(t *testing.T) {
+ tests := []struct {
+ a, b, merged *ResultsEnvelope
+ }{
+ // Run 0: Series that adhere to rule
+ {
+ a: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(10, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10, 0), End: time.Unix(10, 0)},
+ },
+ StepDuration: time.Duration(5) * time.Second,
+ },
+ b: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(5, 0), Value: 1.5},
+ {Timestamp: time.Unix(15, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(5, 0), End: time.Unix(5, 0)},
+ timeseries.Extent{Start: time.Unix(15, 0), End: time.Unix(15, 0)},
+ },
+ StepDuration: time.Duration(5) * time.Second,
+ },
+ merged: &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(5, 0), time.Unix(10, 0), time.Unix(15, 0)},
+ timestamps: map[time.Time]bool{time.Unix(5, 0): true, time.Unix(10, 0): true, time.Unix(15, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(5, 0), Value: 1.5},
+ {Timestamp: time.Unix(10, 0), Value: 1.5},
+ {Timestamp: time.Unix(15, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(5, 0), End: time.Unix(15, 0)},
+ },
+ StepDuration: time.Duration(5) * time.Second,
+ },
+ },
+ // Run 1: Empty second series
+ {
+ a: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(10000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ b: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{},
+ },
+ },
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ merged: &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(10000, 0)},
+ timestamps: map[time.Time]bool{time.Unix(10000, 0): true},
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(10000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ },
+ // Run 2: second series has new metric
+ {
+ a: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(10000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ b: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(15000, 0), End: time.Unix(15000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ merged: &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(10000, 0), time.Unix(15000, 0)},
+ timestamps: map[time.Time]bool{time.Unix(10000, 0): true, time.Unix(15000, 0): true},
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(15000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ },
+ // Run 3: merge one metric, one metric unchanged
+ {
+ a: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(10000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ b: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(15000, 0), End: time.Unix(15000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ merged: &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(10000, 0), time.Unix(15000, 0)},
+ timestamps: map[time.Time]bool{time.Unix(10000, 0): true, time.Unix(15000, 0): true},
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(15000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ },
+ // Run 4: merge multiple extents
+ {
+ a: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(15000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ b: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(30000, 0), Value: 1.5},
+ {Timestamp: time.Unix(35000, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(30000, 0), Value: 1.5},
+ {Timestamp: time.Unix(35000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(30000, 0), End: time.Unix(35000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ merged: &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(10000, 0), time.Unix(15000, 0), time.Unix(30000, 0), time.Unix(35000, 0)},
+ timestamps: map[time.Time]bool{time.Unix(10000, 0): true, time.Unix(15000, 0): true, time.Unix(30000, 0): true, time.Unix(35000, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ {Timestamp: time.Unix(30000, 0), Value: 1.5},
+ {Timestamp: time.Unix(35000, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ {Timestamp: time.Unix(30000, 0), Value: 1.5},
+ {Timestamp: time.Unix(35000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(15000, 0)},
+ timeseries.Extent{Start: time.Unix(30000, 0), End: time.Unix(35000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ },
+ //
+ //
+ // Run 5: merge with some overlapping extents
+ {
+ a: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(15000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ b: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ {Timestamp: time.Unix(20000, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ {Timestamp: time.Unix(20000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(15000, 0), End: time.Unix(20000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ merged: &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(10000, 0), time.Unix(15000, 0), time.Unix(20000, 0)},
+ timestamps: map[time.Time]bool{time.Unix(10000, 0): true, time.Unix(15000, 0): true, time.Unix(20000, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ {Timestamp: time.Unix(20000, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(10000, 0), Value: 1.5},
+ {Timestamp: time.Unix(15000, 0), Value: 1.5},
+ {Timestamp: time.Unix(20000, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(10000, 0), End: time.Unix(20000, 0)},
+ },
+ StepDuration: time.Duration(5000) * time.Second,
+ },
+ },
+ }
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ test.a.Merge(true, test.b)
+ if !reflect.DeepEqual(test.merged, test.a) {
+ t.Errorf("mismatch\n actual=%v\nexpected=%v", test.a, test.merged)
+ }
+ })
+ }
+}
+
+func TestCropToRange(t *testing.T) {
+ tests := []struct {
+ before, after *ResultsEnvelope
+ extent timeseries.Extent
+ }{
+ // Run 0: Case where the very first element in the matrix has a timestamp matching the extent's end
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1644004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1644004600, 0), End: time.Unix(1644004600, 0)},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1644004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1644004600, 0), End: time.Unix(1644004600, 0)},
+ },
+ StepDuration: testStep,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(0, 0),
+ End: time.Unix(1644004600, 0),
+ },
+ },
+ // Run 1: Case where we trim nothing
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1544004600, 0), End: time.Unix(1544004600, 0)},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1544004600, 0), End: time.Unix(1544004600, 0)},
+ },
+ StepDuration: testStep,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(0, 0),
+ End: time.Unix(1644004600, 0),
+ },
+ },
+ // Run 2: Case where we trim everything (all data is too late)
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1544004600, 0), End: time.Unix(1544004600, 0)},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: testStep,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(0, 0),
+ End: time.Unix(10, 0),
+ },
+ },
+ // Run 3: Case where we trim everything (all data is too early)
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(100, 0)},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: testStep,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(10000, 0),
+ End: time.Unix(20000, 0),
+ },
+ },
+ // Run 4: Case where we trim some off the beginning
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "d": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "d": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(300, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(300, 0),
+ End: time.Unix(400, 0),
+ },
+ },
+ // Run 5: Case where we trim some off the ends
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "e": {
+ Metric: map[string]interface{}{"__name__": "e"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "e": {
+ Metric: map[string]interface{}{"__name__": "e"},
+ Points: []Point{
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(200, 0), End: time.Unix(200, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(200, 0),
+ End: time.Unix(200, 0),
+ },
+ },
+ // Run 6: Case where the last datapoint is on the Crop extent
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "f": {
+ Metric: map[string]interface{}{"__name__": "f"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "f": {
+ Metric: map[string]interface{}{"__name__": "f"},
+ Points: []Point{
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(200, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(200, 0),
+ End: time.Unix(300, 0),
+ },
+ },
+ // Run 7: Case where we aren't given any datapoints
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "g": {
+ Metric: map[string]interface{}{"__name__": "g"},
+ Points: []Point{},
+ },
+ },
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(200, 0),
+ End: time.Unix(300, 0),
+ },
+ },
+
+ // Run 9: Case where after cropping, an inner series is empty/removed
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(600, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(400, 0), End: time.Unix(600, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(400, 0),
+ End: time.Unix(600, 0),
+ },
+ },
+ // Run 10: Case where after cropping, the front series is empty/removed
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(600, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(400, 0), End: time.Unix(600, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(400, 0),
+ End: time.Unix(600, 0),
+ },
+ },
+ // Run 11: Case where after cropping, the back series is empty/removed
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(100, 0), Value: 1.5},
+ {Timestamp: time.Unix(200, 0), Value: 1.5},
+ {Timestamp: time.Unix(300, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(100, 0), End: time.Unix(600, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(400, 0), Value: 1.5},
+ {Timestamp: time.Unix(500, 0), Value: 1.5},
+ {Timestamp: time.Unix(600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(400, 0), End: time.Unix(600, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(400, 0),
+ End: time.Unix(600, 0),
+ },
+ },
+ // Run 12: Case where we short circuit since the dataset is already entirely inside the crop range
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{},
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{},
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{},
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(200, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(200, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(100, 0),
+ End: time.Unix(600, 0),
+ },
+ },
+ // Run 13: Case where we short circuit since the dataset is empty
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(200, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(300, 0), End: time.Unix(300, 0)},
+ },
+ StepDuration: time.Duration(100) * time.Second,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(300, 0),
+ End: time.Unix(600, 0),
+ },
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ test.before.CropToRange(test.extent)
+ if !reflect.DeepEqual(test.before, test.after) {
+ t.Errorf("mismatch\nexpected=%v\ngot=%v", test.after, test.before)
+ }
+ })
+ }
+}
+
+const testStep = time.Duration(10) * time.Second
+
+func TestCropToSize(t *testing.T) {
+
+ now := time.Now().Truncate(testStep)
+
+ tests := []struct {
+ before, after *ResultsEnvelope
+ size int
+ bft time.Time
+ extent timeseries.Extent
+ }{
+ // case 0: where we already have the number of timestamps we are cropping to
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1444004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1444004600, 0), End: time.Unix(1444004600, 0)},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1444004600, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1444004600, 0), End: time.Unix(1444004600, 0)},
+ },
+ StepDuration: testStep,
+ timestamps: map[time.Time]bool{time.Unix(1444004600, 0): true},
+ tslist: times.Times{time.Unix(1444004600, 0)},
+ isCounted: true,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(0, 0),
+ End: time.Unix(1444004600, 0),
+ },
+ size: 1,
+ bft: now,
+ },
+
+ // case 1
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1444004600, 0), Value: 1.5},
+ {Timestamp: time.Unix(1444004610, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1444004600, 0), End: time.Unix(1444004610, 0)},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1444004610, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1444004610, 0), End: time.Unix(1444004610, 0)},
+ },
+ StepDuration: testStep,
+ timestamps: map[time.Time]bool{time.Unix(1444004610, 0): true},
+ tslist: times.Times{time.Unix(1444004610, 0)},
+ isCounted: true,
+ isSorted: true,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(0, 0),
+ End: time.Unix(1444004610, 0),
+ },
+ size: 1,
+ bft: now,
+ },
+
+ // case 2 - empty extent list
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{},
+ },
+ },
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{},
+ ExtentList: timeseries.ExtentList{},
+ StepDuration: testStep,
+ },
+ extent: timeseries.Extent{},
+ size: 1,
+ bft: now,
+ },
+
+ // case 3 - backfill tolerance
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1444004610, 0), Value: 1.5},
+ {Timestamp: now, Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1444004610, 0), End: now},
+ },
+ StepDuration: testStep,
+ },
+ after: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1444004610, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1444004610, 0), End: now.Add(-5 * time.Minute)},
+ },
+ StepDuration: testStep,
+ timestamps: map[time.Time]bool{time.Unix(1444004610, 0): true},
+ tslist: times.Times{time.Unix(1444004610, 0)},
+ isCounted: true,
+ isSorted: false,
+ },
+ extent: timeseries.Extent{
+ Start: time.Unix(0, 0),
+ End: now,
+ },
+ size: 2,
+ bft: now.Add(-5 * time.Minute),
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ test.before.CropToSize(test.size, test.bft, test.extent)
+
+ for i := range test.before.ExtentList {
+ test.before.ExtentList[i].LastUsed = time.Time{}
+ }
+
+ if !reflect.DeepEqual(test.before, test.after) {
+ t.Errorf("mismatch\nexpected=%v\n got=%v", test.after, test.before)
+ }
+ })
+ }
+}
+
+func TestUpdateTimestamps(t *testing.T) {
+
+ // test edge condition here (core functionality is tested across this file)
+ re := ResultsEnvelope{isCounted: true}
+ re.updateTimestamps()
+ if re.timestamps != nil {
+ t.Errorf("expected nil map, got size %d", len(re.timestamps))
+ }
+
+}
+
+func TestClone(t *testing.T) {
+
+ tests := []struct {
+ before *ResultsEnvelope
+ }{
+ // Run 0
+ {
+ before: &ResultsEnvelope{
+ Meta: []FieldDefinition{{Name: "1", Type: "string"}},
+ Serializers: map[string]func(interface{}){"test": nil},
+ tslist: times.Times{time.Unix(1644001200, 0)},
+ timestamps: map[time.Time]bool{time.Unix(1644001200, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1644001200, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1644001200, 0), End: time.Unix(1644001200, 0)},
+ },
+ StepDuration: time.Duration(3600) * time.Second,
+ SeriesOrder: []string{"a"},
+ },
+ },
+
+ // Run 1
+ {
+ before: &ResultsEnvelope{
+ tslist: times.Times{time.Unix(1644001200, 0), time.Unix(1644004800, 0)},
+ timestamps: map[time.Time]bool{time.Unix(1644001200, 0): true, time.Unix(1644004800, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1644001200, 0), Value: 1.5},
+ },
+ },
+
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(1644001200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1644004800, 0), Value: 1.5},
+ },
+ },
+ },
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(1644001200, 0), End: time.Unix(1644004800, 0)},
+ },
+ StepDuration: time.Duration(3600) * time.Second,
+ SeriesOrder: []string{"a", "b"},
+ },
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ after := test.before.Clone()
+ if !reflect.DeepEqual(test.before, after) {
+ t.Errorf("mismatch\nexpected %v\nactual %v", test.before, after)
+ }
+ })
+ }
+
+}
+
+func TestSort(t *testing.T) {
+ tests := []struct {
+ before, after *ResultsEnvelope
+ extent timeseries.Extent
+ }{
+ // Case where we trim nothing
+ {
+ before: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004800, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5}, // sort should also dupe kill
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004800, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004800, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ },
+ },
+ },
+ },
+ after: &ResultsEnvelope{
+ isSorted: true,
+ isCounted: true,
+ tslist: []time.Time{time.Unix(1544004000, 0), time.Unix(1544004200, 0), time.Unix(1544004600, 0), time.Unix(1544004800, 0)},
+ timestamps: map[time.Time]bool{time.Unix(1544004000, 0): true, time.Unix(1544004200, 0): true,
+ time.Unix(1544004600, 0): true, time.Unix(1544004800, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004800, 0), Value: 1.5},
+ },
+ },
+ "b": {
+ Metric: map[string]interface{}{"__name__": "b"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004800, 0), Value: 1.5},
+ },
+ },
+ "c": {
+ Metric: map[string]interface{}{"__name__": "c"},
+ Points: []Point{
+ {Timestamp: time.Unix(1544004000, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004200, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004600, 0), Value: 1.5},
+ {Timestamp: time.Unix(1544004800, 0), Value: 1.5},
+ },
+ },
+ },
+ },
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ test.before.isSorted = false
+ test.before.Sort()
+ if !reflect.DeepEqual(test.before, test.after) {
+ t.Errorf("mismatch\nexpected=%v\n actual=%v", test.after, test.before)
+ }
+ // test isSorted short circuit
+ test.before.Sort()
+ if !reflect.DeepEqual(test.before, test.after) {
+ t.Errorf("mismatch\nexpected=%v\n actual=%v", test.after, test.before)
+ }
+ })
+ }
+}
+
+func TestSetExtents(t *testing.T) {
+ re := &ResultsEnvelope{}
+ ex := timeseries.ExtentList{timeseries.Extent{Start: time.Time{}, End: time.Time{}}}
+ re.SetExtents(ex)
+ if len(re.ExtentList) != 1 {
+ t.Errorf(`expected 1. got %d`, len(re.ExtentList))
+ }
+}
+
+func TestExtents(t *testing.T) {
+ re := &ResultsEnvelope{}
+ ex := timeseries.ExtentList{timeseries.Extent{Start: time.Time{}, End: time.Time{}}}
+ re.SetExtents(ex)
+ e := re.Extents()
+ if len(e) != 1 {
+ t.Errorf(`expected 1. got %d`, len(re.ExtentList))
+ }
+}
+
+func TestSeriesCount(t *testing.T) {
+ re := &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "d": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(99, 0), Value: 1.5},
+ {Timestamp: time.Unix(199, 0), Value: 1.5},
+ {Timestamp: time.Unix(299, 0), Value: 1.5},
+ },
+ },
+ },
+ }
+ if re.SeriesCount() != 1 {
+ t.Errorf("expected 1 got %d.", re.SeriesCount())
+ }
+}
+
+func TestValueCount(t *testing.T) {
+ re := &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "d": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(99, 0), Value: 1.5},
+ {Timestamp: time.Unix(199, 0), Value: 1.5},
+ {Timestamp: time.Unix(299, 0), Value: 1.5},
+ },
+ },
+ },
+ }
+ if re.ValueCount() != 3 {
+ t.Errorf("expected 3 got %d.", re.ValueCount())
+ }
+}
+
+func TestTimestampCount(t *testing.T) {
+
+ tests := []struct {
+ ts *ResultsEnvelope
+ expected int
+ }{
+ {
+ ts: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "d": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(99, 0), Value: 1.5},
+ {Timestamp: time.Unix(199, 0), Value: 1.5},
+ {Timestamp: time.Unix(299, 0), Value: 1.5},
+ },
+ },
+ },
+ },
+ expected: 3,
+ },
+
+ {
+ ts: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "d": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(99, 0), Value: 1.5},
+ {Timestamp: time.Unix(199, 0), Value: 1.5},
+ },
+ },
+ },
+ },
+ expected: 2,
+ },
+
+ {
+ ts: &ResultsEnvelope{
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "d"},
+ Points: []Point{
+ {Timestamp: time.Unix(99, 0), Value: 1.5},
+ {Timestamp: time.Unix(199, 0), Value: 1.5},
+ },
+ },
+ "e": {
+ Metric: map[string]interface{}{"__name__": "e"},
+ Points: []Point{
+ {Timestamp: time.Unix(99, 0), Value: 1.5},
+ {Timestamp: time.Unix(299, 0), Value: 1.5},
+ },
+ },
+ },
+ },
+ expected: 3,
+ },
+ }
+
+ for i, test := range tests {
+ t.Run(strconv.Itoa(i), func(t *testing.T) {
+ tc := test.ts.TimestampCount()
+ if tc != test.expected {
+ t.Errorf("expected %d got %d.", test.expected, tc)
+ }
+ })
+ }
+}
+
+func TestMergeSeriesOrder(t *testing.T) {
+
+ re := ResultsEnvelope{}
+ so1 := []string{"a", "e"}
+ re.mergeSeriesOrder(so1)
+ if !reflect.DeepEqual(re.SeriesOrder, so1) {
+ t.Errorf("expected [%s] got [%s]", strings.Join(so1, ","), strings.Join(re.SeriesOrder, ","))
+ }
+ re.Data = map[string]*DataSet{"a": nil, "e": nil}
+
+ so2 := []string{"d", "e"}
+ ex2 := []string{"a", "d", "e"}
+ re.mergeSeriesOrder(so2)
+ if !reflect.DeepEqual(re.SeriesOrder, ex2) {
+ t.Errorf("expected [%s] got [%s]", strings.Join(ex2, ","), strings.Join(re.SeriesOrder, ","))
+ }
+ re.Data = map[string]*DataSet{"a": nil, "d": nil, "e": nil}
+
+ so3 := []string{"b", "c", "e"}
+ ex3 := []string{"a", "d", "b", "c", "e"}
+ re.mergeSeriesOrder(so3)
+ if !reflect.DeepEqual(re.SeriesOrder, ex3) {
+ t.Errorf("expected [%s] got [%s]", strings.Join(ex3, ","), strings.Join(re.SeriesOrder, ","))
+ }
+ re.Data = map[string]*DataSet{"a": nil, "d": nil, "b": nil, "c": nil, "e": nil}
+
+ so4 := []string{"f"}
+ ex4 := []string{"a", "d", "b", "c", "e", "f"}
+ re.mergeSeriesOrder(so4)
+ if !reflect.DeepEqual(re.SeriesOrder, ex4) {
+ t.Errorf("expected [%s] got [%s]", strings.Join(ex4, ","), strings.Join(re.SeriesOrder, ","))
+ }
+
+}
+
+func TestSize(t *testing.T) {
+ r := &ResultsEnvelope{
+ isCounted: true,
+ isSorted: true,
+ tslist: times.Times{time.Unix(5, 0), time.Unix(10, 0), time.Unix(15, 0)},
+ timestamps: map[time.Time]bool{time.Unix(5, 0): true, time.Unix(10, 0): true, time.Unix(15, 0): true},
+ Data: map[string]*DataSet{
+ "a": {
+ Metric: map[string]interface{}{"__name__": "a"},
+ Points: []Point{
+ {Timestamp: time.Unix(5, 0), Value: 1.5},
+ {Timestamp: time.Unix(10, 0), Value: 1.5},
+ {Timestamp: time.Unix(15, 0), Value: 1.5},
+ },
+ },
+ },
+ Meta: []FieldDefinition{{Name: "test", Type: "Test"}},
+ SeriesOrder: []string{"test"},
+ ExtentList: timeseries.ExtentList{
+ timeseries.Extent{Start: time.Unix(5, 0), End: time.Unix(15, 0)},
+ },
+ StepDuration: time.Duration(5) * time.Second,
+ }
+ i := r.Size()
+ const expected = 146
+ if i != expected {
+ t.Errorf("expected %d got %d", expected, i)
+ }
+}
diff --git a/internal/proxy/origins/clickhouse/stubs.go b/internal/proxy/origins/clickhouse/stubs.go
new file mode 100644
index 000000000..bc849dedc
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/stubs.go
@@ -0,0 +1,41 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "net/http"
+ "net/url"
+
+ "github.com/Comcast/trickster/internal/timeseries"
+)
+
+// This file holds funcs required by the Proxy Client or Timeseries interfaces,
+// but are (currently) unused by the ClickHouse implementation.
+
+// Series (timeseries.Timeseries Interface) stub funcs
+
+// FastForwardURL is not used for ClickHouse and is here to conform to the Proxy Client interface
+func (c *Client) FastForwardURL(r *http.Request) (*url.URL, error) {
+ return nil, nil
+}
+
+// ClickHouse Client (proxy.Client Interface) stub funcs
+
+// UnmarshalInstantaneous is not used for ClickHouse and is here to conform to the Proxy Client interface
+func (c *Client) UnmarshalInstantaneous(data []byte) (timeseries.Timeseries, error) {
+ return nil, nil
+}
+
+// QueryRangeHandler is not used for ClickHouse and is here to conform to the Proxy Client interface
+func (c *Client) QueryRangeHandler(w http.ResponseWriter, r *http.Request) {}
diff --git a/internal/proxy/origins/clickhouse/stubs_test.go b/internal/proxy/origins/clickhouse/stubs_test.go
new file mode 100644
index 000000000..1f924e131
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/stubs_test.go
@@ -0,0 +1,51 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "testing"
+)
+
+func TestFastForwardURL(t *testing.T) {
+
+ client := &Client{}
+ u, err := client.FastForwardURL(nil)
+ if u != nil {
+ t.Errorf("Expected nil url, got %s", u)
+ }
+
+ if err != nil {
+ t.Errorf("Expected nil err, got %s", err)
+ }
+}
+
+func TestUnmarshalInstantaneous(t *testing.T) {
+
+ client := &Client{}
+ tr, err := client.UnmarshalInstantaneous(nil)
+
+ if tr != nil {
+ t.Errorf("Expected nil timeseries, got %s", tr)
+ }
+
+ if err != nil {
+ t.Errorf("Expected nil err, got %s", err)
+ }
+
+}
+
+func TestQueryRangeHandler(t *testing.T) {
+ client := &Client{}
+ client.QueryRangeHandler(nil, nil)
+}
diff --git a/internal/proxy/origins/clickhouse/tokenization.go b/internal/proxy/origins/clickhouse/tokenization.go
new file mode 100644
index 000000000..84285e776
--- /dev/null
+++ b/internal/proxy/origins/clickhouse/tokenization.go
@@ -0,0 +1,138 @@
+/**
+* Copyright 2018 Comcast Cable Communications Management, LLC
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+* http://www.apache.org/licenses/LICENSE-2.0
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+ */
+
+package clickhouse
+
+import (
+ "fmt"
+ "regexp"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/Comcast/trickster/internal/timeseries"
+ "github.com/Comcast/trickster/internal/util/regexp/matching"
+)
+
+// This file handles tokenization of time parameters within ClickHouse queries
+// for cache key hashing and delta proxy caching.
+
+// Tokens for String Interpolation
+const (
+ tkTimestamp1 = "<$TIMESTAMP1$>"
+ tkTimestamp2 = "<$TIMESTAMP2$>"
+)
+
+var reTimeFieldAndStep, reTimeClauseAlt *regexp.Regexp
+
+func init() {
+ reTimeFieldAndStep = regexp.MustCompile(`(?i)select\s+\(\s*intdiv\s*\(\s*touint32\s*\(\s*(?P