Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add cacheStore for hstream #1828

Merged
merged 36 commits into from
Jul 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
125621c
Basic implementation of CacheStore
YangKian Jun 12, 2024
4677e03
refactor cache store
YangKian Jun 13, 2024
6c37ea2
implement health checker for hstore
YangKian Jun 17, 2024
48d0c20
refactor cacheStore: split read/write mode for cacheStore && add retr…
YangKian Jun 17, 2024
9c33981
fix cacheStore: make sure key is unique
YangKian Jun 19, 2024
448769e
code clean for hstore checker
YangKian Jun 19, 2024
a28aab2
implement health monitor
YangKian Jun 19, 2024
adcc752
handle lookup when meta cluster unavailable
YangKian Jun 19, 2024
1de7b0b
healthMonitor: update cabal file
YangKian Jun 19, 2024
23dc65d
add config
YangKian Jun 19, 2024
41c0df9
integration
YangKian Jun 19, 2024
d0b4009
cache store: fix encode key to make sure record write in order
YangKian Jun 26, 2024
f7c9df9
change ld_checker ffi to safe
YangKian Jun 26, 2024
b90ab56
every time server switch to backup mode, recreate column family itera…
YangKian Jun 26, 2024
73339b8
fix open db
YangKian Jun 26, 2024
71cfb2d
update
YangKian Jun 26, 2024
390b6a2
add more rocksdb options
YangKian Jun 26, 2024
f8f9c9f
bypass reader when server is in backup mode
YangKian Jun 26, 2024
98fbed0
log update
YangKian Jun 28, 2024
7e975b9
add HStreamEnableCacheStore flag
YangKian Jun 28, 2024
2ae14c6
Revert "add HStreamEnableCacheStore flag"
YangKian Jul 1, 2024
e924d36
add cli option to enable server cache
YangKian Jul 1, 2024
350bc1e
add cache store related stats
YangKian Jul 1, 2024
1945152
add healthy monitor related stats
YangKian Jul 2, 2024
35e0cb7
add per_cache_store_stats.inc
YangKian Jul 2, 2024
d9c4318
update stats handler
YangKian Jul 2, 2024
4e6a6a4
change writeRecord method to return an Either result
YangKian Jul 2, 2024
056c4be
fix stats
YangKian Jul 3, 2024
2c5e425
catch meta exception when get ConnectorStatsIsAlive stat
YangKian Jul 3, 2024
3df16e7
fix stream append stats
YangKian Jul 3, 2024
9dd6646
update cache store stats
YangKian Jul 3, 2024
b9e3173
update protocol
YangKian Jul 3, 2024
7f94aaf
disable unsupported rocksdb option
YangKian Jul 3, 2024
7e853c8
fix test
YangKian Jul 4, 2024
f5599c4
update
YangKian Jul 4, 2024
2f10a09
handle rocksdb exception in write path
YangKian Jul 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update stats handler
  • Loading branch information
YangKian committed Jul 2, 2024
commit d9c4318015ce4f79411b5377d1590ee6d168e40e
26 changes: 26 additions & 0 deletions hstream/src/HStream/Server/Handler/Stats.hs
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,8 @@ getStatsInternal ServerContext{scStatsHolder = holder} s@(StatTypeStatQueryStat
getQueryStatsInternal holder stats <&> convert s
getStatsInternal ServerContext{scStatsHolder = holder} s@(StatTypeStatViewStat stats) = do
getViewStatsInternal holder stats <&> convert s
getStatsInternal ServerContext{scStatsHolder = holder} s@(StatTypeStatCacheStoreStat stats) = do
getCacheStoreStatsInternal holder stats <&> convert s

getStreamStatsInternal
:: Stats.StatsHolder
Expand Down Expand Up @@ -209,6 +211,30 @@ getConnectorStatsInternal statsHolder ioWorker (PS.Enumerated stats) = do
map (\API.Connector{..} -> if connectorStatus == "RUNNING" then (U.textToCBytes connectorName, 1) else (U.textToCBytes connectorName, 0)) cs
Left _ -> return . Left . T.pack $ "invalid stat type " <> show stats

getCacheStoreStatsInternal
:: Stats.StatsHolder
-> PS.Enumerated API.CacheStoreStats
-> IO (Either T.Text (Map CBytes Int64))
getCacheStoreStatsInternal statsHolder (PS.Enumerated stats) = do
Log.debug $ "request stream stats: " <> Log.buildString' stats
s <- Stats.newAggregateStats statsHolder
case stats of
Right API.CacheStoreStatsCSAppendInBytes ->
Stats.cache_store_stat_getall_append_in_bytes s <&> Right
Right API.CacheStoreStatsCSAppendInRecords ->
Stats.cache_store_stat_getall_append_in_records s <&> Right
Right API.CacheStoreStatsCSAppendTotal ->
Stats.cache_store_stat_getall_append_total s <&> Right
Right API.CacheStoreStatsCSAppendFailed ->
Stats.cache_store_stat_getall_append_failed s <&> Right
Right API.CacheStoreStatsCSReadInBytes ->
Stats.cache_store_stat_getall_read_in_bytes s <&> Right
Right API.CacheStoreStatsCSReadInRecords ->
Stats.cache_store_stat_getall_read_in_records s <&> Right
Right API.CacheStoreStatsCSDeliveredInRecords ->
Stats.cache_store_stat_getall_delivered_in_records s <&> Right
Left _ -> return . Left . T.pack $ "invalid stat type " <> show stats

getQueryStatsInternal
:: Stats.StatsHolder
-> PS.Enumerated API.QueryStats
Expand Down