Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shard query splitting: Test in CI #747

Closed
wants to merge 41 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
e6cd452
feat(CombineResponses): create module
matyax Aug 16, 2024
56786a6
test(CombineResponses): add unit tests and adjust implementation
matyax Aug 16, 2024
c2018b0
chore: install uuid
matyax Aug 16, 2024
c77b15a
feat(shardQuerySplitting): create module
matyax Aug 16, 2024
385dc2c
chore: interpolate queries
matyax Aug 16, 2024
d97a77c
feat(shardQuerySplitting): query from high to low shard
matyax Aug 16, 2024
698dffb
fix(combineResponses): improve nanos handling
matyax Aug 16, 2024
fa51bef
feat(shardQuerySplitting): group shards in requests
matyax Aug 19, 2024
58866f8
feat(mergeResponses): properly calculate dataframe length
matyax Aug 19, 2024
ac6715e
feat(shardQuerySplitting): retry failed requests
matyax Aug 20, 2024
60e2f33
feat(mergeResponses): compare timestamps using nanoseconds if present
matyax Aug 21, 2024
7577500
test(combineResponses): add extra test case
matyax Aug 21, 2024
3adc6a8
feat(mergeResponses): fix frame length calculation
matyax Aug 21, 2024
d745d47
feat(shardQuerySplitting): sort shards desc
matyax Aug 22, 2024
234ea84
fix(shardQuerySplitting): increase retries
matyax Aug 22, 2024
1f92fc2
feat(shardQuerySplitting): adjust shards based on timerange
matyax Aug 22, 2024
765664a
chore(ci): simplify
matyax Aug 22, 2024
bbd4be5
feat(shardQuerySplitting): dynamically calculate max requests
matyax Aug 26, 2024
f4f2a9f
fix(shardQuerySplitting): do not rely on max shard as shards length
matyax Aug 26, 2024
94a596f
feat(shardQuerySplitting): pass service name to values call
matyax Aug 27, 2024
fcf2097
feat(shardQuerySplitting): do not retry if maximum series reached
matyax Aug 27, 2024
0234c31
feat(combineResponses): consider id when comparing log timestamps
matyax Aug 28, 2024
feb84d8
chore: print shards and not max shard
matyax Aug 28, 2024
1807e47
chore: remove max shard
matyax Aug 28, 2024
ed73c8b
fix(combineResponses): fix nanos recombination
matyax Aug 28, 2024
64fb778
chore: words
matyax Aug 29, 2024
5c69dad
feat(querying): exclude log queries from sharding
matyax Aug 29, 2024
07bdc62
fix(getShardRequests): debug group generation
matyax Aug 29, 2024
1b9e1b6
feat(shardQuerySplitting): integrate feature flag
matyax Aug 29, 2024
b8d096e
feat(shardQuerySplitting): fix falling back to regular query
matyax Sep 2, 2024
bf39ccc
chore: rename functions
matyax Sep 2, 2024
a3b9d9b
feat(jest): add observable matchers
matyax Sep 3, 2024
79727ff
test(shardQuerySplitting): add base test setup
matyax Sep 3, 2024
240aab0
test(shardQuerySplitting): add more tests
matyax Sep 3, 2024
b72a325
Revert "chore(ci): simplify"
matyax Sep 4, 2024
c089d09
chore: spelling
matyax Sep 4, 2024
89e6114
chore(shardQuerySplitting): add docs
matyax Sep 4, 2024
99796a7
chore: simplify
matyax Sep 4, 2024
6ba7fd7
chore: remove word
matyax Sep 4, 2024
c0df8ca
chore: fix test title
matyax Sep 4, 2024
aa230cd
test(shardQuerySplitting): enable feature and test e2e
matyax Sep 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions index.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { Observable } from 'rxjs';

type ObservableType<T> = T extends Observable<infer V> ? V : never;

declare global {
namespace jest {
interface Matchers<R, T = {}> {
toEmitValues<E = ObservableType<T>>(expected: E[]): Promise<CustomMatcherResult>;
/**
* Collect all the values emitted by the observables (also errors) and pass them to the expectations functions after
* the observable ended (or emitted error). If Observable does not complete within OBSERVABLE_TEST_TIMEOUT_IN_MS the
* test fails.
*/
toEmitValuesWith<E = ObservableType<T>>(expectations: (received: E[]) => void): Promise<CustomMatcherResult>;
}
}
}
6 changes: 6 additions & 0 deletions jest-setup.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,15 @@ import { TextDecoder, TextEncoder } from 'util';

import './.config/jest-setup';

import { toEmitValuesWith } from './tests/matchers';

global.TextEncoder = TextEncoder;
global.TextDecoder = TextDecoder;

expect.extend({
toEmitValuesWith,
});

// mock the intersection observer and just say everything is in view
const mockIntersectionObserver = jest.fn().mockImplementation((callback) => ({
observe: jest.fn().mockImplementation((elem) => {
Expand Down
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
"@types/react-scroll-sync": "^0.9.0",
"@types/react-table": "^7.7.20",
"@types/testing-library__jest-dom": "5.14.8",
"@types/uuid": "^10.0.0",
"copy-webpack-plugin": "^11.0.0",
"cspell": "6.13.3",
"css-loader": "^6.7.3",
Expand Down Expand Up @@ -93,7 +94,8 @@
"react-dom": "18.2.0",
"react-router-dom": "^5.2.0",
"rxjs": "7.8.1",
"tslib": "2.5.3"
"tslib": "2.5.3",
"uuid": "^10.0.0"
},
"resolutions": {
"**/fast-loops": "^1.1.4"
Expand Down
8 changes: 8 additions & 0 deletions project-words.txt
Original file line number Diff line number Diff line change
Expand Up @@ -473,3 +473,11 @@ REFID
unroute
myown
yass
decbytes
frametype
Nanos
nanos
subquery
Subscription
uuidv
without
Loading
Loading