tree: fix vectorized COALESCE and NULLIF type checking with VOID #83868

msirek · 2022-07-05T23:11:46Z

Previously, the COALESCE and NULLIF operators could error out in
vectorized execution when comparing a VOID datum with NULL.

This occurred due to the unique property of a VOID, that it
can't be compared with another VOID using any of the normal operators
such as =, <, >..., or even IS [NOT] [DISTINCT FROM], for example:

SELECT ''::VOID IS DISTINCT FROM ''::VOID;
ERROR: unsupported comparison operator: <void> IS DISTINCT FROM <void>
SQLSTATE: 22023

Processing of COALESCE in the columnar execution engine for an
expression like COALESCE(void_col, NULL) builds an IS DISTINCT FROM
operation between the VOID column and NULL here:

cockroach/pkg/sql/colexec/colbuilder/execplan.go

Lines 2122 to 2129 in ea559df

    
           whens[i] = &tree.When{ 
        
           	Cond: tree.NewTypedComparisonExpr( 
        
           		treecmp.MakeComparisonOperator(treecmp.IsDistinctFrom), 
        
           		t.Exprs[i].(tree.TypedExpr), 
        
           		tree.DNull, 
        
           	), 
        
           	Val: t.Exprs[i], 
        
           }

This comparison is assumed to be OK, but fails with an internal error
because comparison with UnknownType is only implicitly supported if
the type can be compared with itself.

To address this, this patch modifies vectorized COALESCE to use the
IS NOT NULL operator internally instead of IS DISTINCT FROM whenever
the latter is not a valid comparison.
A similar problem with NULLIF, which internally uses =, is fixed.
Type checking of NULLIF is enhanced in the parser so incompatible
comparisons can be caught prior to execution time.

Release note (bug fix): This patch fixes vectorized evaluation of
COALESCE involving expressions of type VOID and enhances type checking
of NULLIF expressions with VOID, so incompatible comparisons can
be caught during query compilation instead of during query execution.

cockroach-teamcity · 2022-07-05T23:11:55Z

This change is

yuzefovich

Reviewed all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @msirek, and @otan)

pkg/sql/logictest/testdata/logic_test/void line 76 at r1 (raw file):


statement ok
SET vectorize=off

nit: you should do something like this:

SET vectorize=on

<query>

SET vectorize=off

< query>

RESET vectorize

when you want to control the vectorize mode precisely, compare the results for two engines, and so that the setup is left unchanged in the end.

As it is written currently, the first execution can happen with any vectorize mode (depending on the config), the second one is guaranteed to be vectorize=off, and vectorize=off remains for all configs for the rest of the file (should new queries are added to the file).

otan · 2022-07-05T23:20:49Z

pkg/sql/sem/tree/eval.go

@@ -1651,6 +1651,8 @@ var CmpOps = cmpOpFixups(map[treecmp.ComparisonOperatorSymbol]cmpOpOverload{
 		makeIsFn(types.TimestampTZ, types.Timestamp, volatility.Stable),
 		makeIsFn(types.Time, types.TimeTZ, volatility.Stable),
 		makeIsFn(types.TimeTZ, types.Time, volatility.Stable),
+		makeIsFn(types.Unknown, types.Void, volatility.LeakProof),


can we fix type resolution to instead evaluate NULL as a VOID? i wonder if this also gets fixed by adding void here (it might not)

cockroach/pkg/sql/sem/tree/constant.go

Line 480 in c48196c

StrValAvailAllParsable = []*types.T{

msirek

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @msirek, and @otan)

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

can we fix type resolution to instead evaluate NULL as a VOID? i wonder if this also gets fixed by adding void here (it might not)

cockroach/pkg/sql/sem/tree/constant.go

Line 480 in c48196c

StrValAvailAllParsable = []*types.T{

If I understand correctly, you're suggesting adding types.Void, at the end of this slice. This does not fix the error, and the code in eval.go is still required.

And, adding this would mean that type annotation on any string, such as 'foo':::VOID would now succeed. Is this what we want?
Support for type annotation of the empty string to VOID is already added through #82965.

otan · 2022-07-05T23:55:17Z

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, msirek (Mark Sirek) wrote…

If I understand correctly, you're suggesting adding types.Void, at the end of this slice. This does not fix the error, and the code in eval.go is still required.

And, adding this would mean that type annotation on any string, such as 'foo':::VOID would now succeed. Is this what we want?
Support for type annotation of the empty string to VOID is already added through #82965.

i guess why do we special case Unknown for void but not the others? Ideally this comparison is fixed during type checking

otan · 2022-07-05T23:55:47Z

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

i guess why do we special case Unknown for void but not the others? Ideally this comparison is fixed during type checking

Sorry by "comparison is fixed" i mean "comparison is made and evaluated to be a void comparison"

msirek

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @msirek, and @otan)

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

i guess why do we special case Unknown for void but not the others?

Because all other types are equivalent to each other, causing this line to be true for each of the inputs:

cockroach/pkg/sql/sem/tree/overload.go

Line 298 in 94fe056

    
           return i < len(a) && (typ.Family() == types.UnknownFamily || a[i].Typ.Equivalent(typ))

a[i].Typ.Equivalent(typ) is false when a[i].Typ is void and type is void. But this is by design:

select ''::VOID = ''::VOID;
ERROR: unsupported comparison operator: <void> = <void>
SQLSTATE: 22023

So, VOID is just a special type.

msirek

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @otan, and @yuzefovich)

pkg/sql/logictest/testdata/logic_test/void line 76 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: you should do something like this:
SET vectorize=on

<query>

SET vectorize=off

< query>

RESET vectorize
when you want to control the vectorize mode precisely, compare the results for two engines, and so that the setup is left unchanged in the end.

As it is written currently, the first execution can happen with any vectorize mode (depending on the config), the second one is guaranteed to be vectorize=off, and vectorize=off remains for all configs for the rest of the file (should new queries are added to the file).

Fixed it, thanks!

otan · 2022-07-06T00:06:41Z

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, msirek (Mark Sirek) wrote…

i guess why do we special case Unknown for void but not the others?

Because all other types are equivalent to each other, causing this line to be true for each of the inputs:

cockroach/pkg/sql/sem/tree/overload.go

Line 298 in 94fe056

return i < len(a) && (typ.Family() == types.UnknownFamily || a[i].Typ.Equivalent(typ))

a[i].Typ.Equivalent(typ) is false when a[i].Typ is void and type is void. But this is by design:
select ''::VOID = ''::VOID;
ERROR: unsupported comparison operator: <void> = <void>
SQLSTATE: 22023
So, VOID is just a special type.

interesting... select ''::VOID = ''::VOID; is equating the values of the types though, not the types itself.
i'd argue types.VoidFamily.Equivalent(types.VoidFamily) should be true...

otan · 2022-07-06T00:10:51Z

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

interesting... select ''::VOID = ''::VOID; is equating the values of the types though, not the types itself.
i'd argue types.VoidFamily.Equivalent(types.VoidFamily) should be true...

i think this may stem from the fact that IS is different to =.

In pg:

otan=# select ''::void = null::void;
ERROR:  operator does not exist: void = void
LINE 1: select ''::void = null::void;

which is different to Is:

otan=# select ''::void IS null;
 ?column?
----------
 f
(1 row)

so maybe adding this makeIsFn is necessary anyway as fixing type resolution to recognise the NULL case is different.

It may be worth adding a comment about why this is special cased. Sorry for making a seemingly big deal about two lines of code, I really just wanted to dig into the special case!

msirek

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @otan, and @yuzefovich)

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

select ''::VOID = ''::VOID; is equating the values of the types though, not the types itself.

Yes, I know, but in the process of testing if the = operator can be applied we check if the types are equivalent. This applies to any of the operators, such as IS DISTINCT FROM, which is the operator we care about here.

which is different to Is:
otan=# select ''::void IS null;

This example is comparing VoidType with UnknownType, not VoidType with VoidType.

i'd argue types.VoidFamily.Equivalent(types.VoidFamily) should be true...

If that were the case, then we'd have different behavior from postgres, which treats them not as equivalent:

postgres=# select ''::void is distinct from ''::void;
ERROR:  operator does not exist: void = void
LINE 1: select ''::void is distinct from ''::void;
                        ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
postgres=#

otan

please add a comment as per comment below

msirek

Added:

		// Void is unique in that it is not equivalent with itself, so implicit
		// equivalence with Unknown in function ArgTypes.MatchAt due to the check
		// `(typ.Family() == types.UnknownFamily || a[i].Typ.Equivalent(typ))` does
		// not occur. Therefore, to allow the comparison
		// `''::VOID IS DISTINCT FROM NULL`, an explicit equivalence with Unknown is
		// added:
		makeIsFn(types.Unknown, types.Void, volatility.LeakProof),
		makeIsFn(types.Void, types.Unknown, volatility.LeakProof),

Do you have an opinion on whether we should support type annotation to VOID on any string to match CAST?
Currently only '':::VOID is legal, but since 'foo'::VOID is legal, should 'foo':::VOID also be legal? In #82965 we decided to only allow '':::VOID since that's what SQLSmith needed, and '' is a valid string representation of a VOID literal, but perhaps type annotation should have the same behavior as CAST.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @otan, and @yuzefovich)

msirek

bors r+

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @otan, and @yuzefovich)

craig · 2022-07-08T21:18:48Z

Build failed:

GitHub CI (Cockroach)

msirek · 2022-07-08T23:48:22Z

bors r+

craig · 2022-07-09T00:14:52Z

Build failed (retrying...):

GitHub CI (Cockroach)

msirek · 2022-07-09T00:17:31Z

bors r-

craig · 2022-07-09T00:17:33Z

Canceled.

mgartner · 2022-07-11T13:39:24Z

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, msirek (Mark Sirek) wrote…

select ''::VOID = ''::VOID; is equating the values of the types though, not the types itself.

Yes, I know, but in the process of testing if the = operator can be applied we check if the types are equivalent. This applies to any of the operators, such as IS DISTINCT FROM, which is the operator we care about here.

which is different to Is:
otan=# select ''::void IS null;

This example is comparing VoidType with UnknownType, not VoidType with VoidType.

i'd argue types.VoidFamily.Equivalent(types.VoidFamily) should be true...

If that were the case, then we'd have different behavior from postgres, which treats them not as equivalent:
postgres=# select ''::void is distinct from ''::void;
ERROR:  operator does not exist: void = void
LINE 1: select ''::void is distinct from ''::void;
                        ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
postgres=# 

I'm not convinced that PG supports the [VOID] IS DISTINCT FROM [UNKNOWN] operator:

marcus=# select ''::void is distinct from null::unknown;
ERROR:  42883: operator does not exist: void = unknown
LINE 1: select ''::void is distinct from null::unknown;
                        ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
LOCATION:  op_error, parse_oper.c:656

When NULL is explicitly cast to the unknown type, the comparison fails. Which raises the question: which operator overload is being used for ''::VOID IS DISTINCT FROM NULL... Maybe this inconsistent behavior is a bug in Postgres?

msirek

@mgartner @yuzefovich Please take a look at the new fix.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @otan, and @yuzefovich)

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

I'm not convinced that PG supports the [VOID] IS DISTINCT FROM [UNKNOWN] operator:

You're right! I found they're normalizing this to an IS NOT NULL operator when one side is NULL. The code only looks for constant NULL, not a type-cast, so is using IS DISTINCT FROM when the cast is present.
The root problem for this case is COALESCE uses IS DISTINCT FROM when vectorized, but in the row engine evaluates each expression in the COALESCE directly into a Datum, then compares that Datum directly in Go code with DNull:

cockroach/pkg/sql/sem/eval/expr.go

Lines 187 to 194 in 5a9ec7c

    
           func (e *evaluator) EvalCoalesceExpr(expr *tree.CoalesceExpr) (tree.Datum, error) { 
        
           	for _, ex := range expr.Exprs { 
        
           		d, err := ex.(tree.TypedExpr).Eval(e) 
        
           		if err != nil { 
        
           			return nil, err 
        
           		} 
        
           		if d != tree.DNull { 
        
           			return d, nil

It's now fixed by using IS NOT NULL in place of IS DISTINCT FROM NULL.
PTAL

yuzefovich

but it's worth to get another approval too.

Reviewed 6 of 6 files at r4, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @mgartner and @otan)

mgartner · 2022-07-12T17:26:15Z

pkg/sql/colexec/colbuilder/execplan.go line 2128 at r5 (raw file):

		for i := range whens {
			whens[i] = &tree.When{
				Cond: tree.NewTypedIsNotNullExpr(t.Exprs[i].(tree.TypedExpr)),

I hate to throw a wrench in this, but x IS NOT NULL and x IS DISTINCT FROM NULL are not equivalent when x is a tuple (learned this lesson in #48299). In Postgres:

marcus=# SELECT (NULL, NULL) IS NOT NULL;
 ?column?
----------
 f
(1 row)

marcus=# SELECT (NULL, NULL) IS DISTINCT FROM NULL;
 ?column?
----------
 t
(1 row)

So we can't make this transformation in all cases. A good test case to add would be the example in PG below. Notice how (NULL, NULL) is returned.

marcus=# SELECT COALESCE((NULL, NULL), (1, 2));
 coalesce
----------
 (,)
(1 row)

An alternative would be to use IS NOT NULL only if t.Exprs[i] is type VOID, and use IS DISTINCT FROM otherwise.

Or, since we know that any value of type VOID (or the singular value? or the empty value?) is always distinct from NULL, can we simply fold the coalesce into t.Exprs[i] when it is of type void? The optimizer could also make this transformation in a normalization rule.

Maybe there is a better idea if we think those alternatives are too hacky.

mgartner · 2022-07-12T17:26:46Z

pkg/sql/sem/tree/eval.go line 1654 at r1 (raw file):

Previously, msirek (Mark Sirek) wrote…

I'm not convinced that PG supports the [VOID] IS DISTINCT FROM [UNKNOWN] operator:

You're right! I found they're normalizing this to an IS NOT NULL operator when one side is NULL. The code only looks for constant NULL, not a type-cast, so is using IS DISTINCT FROM when the cast is present.
The root problem for this case is COALESCE uses IS DISTINCT FROM when vectorized, but in the row engine evaluates each expression in the COALESCE directly into a Datum, then compares that Datum directly in Go code with DNull:

cockroach/pkg/sql/sem/eval/expr.go

Lines 187 to 194 in 5a9ec7c

func (e *evaluator) EvalCoalesceExpr(expr *tree.CoalesceExpr) (tree.Datum, error) {

for _, ex := range expr.Exprs {

d, err := ex.(tree.TypedExpr).Eval(e)

if err != nil {

return nil, err

}

if d != tree.DNull {

return d, nil

It's now fixed by using IS NOT NULL in place of IS DISTINCT FROM NULL.
PTAL

Thanks for investigating deeper!

msirek

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @mgartner, @otan, and @yuzefovich)

pkg/sql/colexec/colbuilder/execplan.go line 2128 at r5 (raw file):

Previously, mgartner (Marcus Gartner) wrote…

I hate to throw a wrench in this, but x IS NOT NULL and x IS DISTINCT FROM NULL are not equivalent when x is a tuple (learned this lesson in #48299). In Postgres:
marcus=# SELECT (NULL, NULL) IS NOT NULL;
 ?column?
----------
 f
(1 row)

marcus=# SELECT (NULL, NULL) IS DISTINCT FROM NULL;
 ?column?
----------
 t
(1 row)
So we can't make this transformation in all cases. A good test case to add would be the example in PG below. Notice how (NULL, NULL) is returned.
marcus=# SELECT COALESCE((NULL, NULL), (1, 2));
 coalesce
----------
 (,)
(1 row)
An alternative would be to use IS NOT NULL only if t.Exprs[i] is type VOID, and use IS DISTINCT FROM otherwise.

Or, since we know that any value of type VOID (or the singular value? or the empty value?) is always distinct from NULL, can we simply fold the coalesce into t.Exprs[i] when it is of type void? The optimizer could also make this transformation in a normalization rule.

Maybe there is a better idea if we think those alternatives are too hacky.

Thanks for catching this! I'm not sure what the 100% best option is, but I've updated the fix to use IS DISTINCT FROM if it can (by checking if the type is compatible with comparison to NULL), and if not, IS NOT NULL. VOID should be the only type this applies to, but if another type is added which is similar to VOID, this should automatically handle it too.

rytaft · 2022-07-18T13:21:35Z

Can we get this merged? @mgartner do you still need to sign off?

mgartner

Reviewed 1 of 5 files at r5, 3 of 3 files at r6, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (and 1 stale) (waiting on @msirek, @otan, and @yuzefovich)

-- commits line 27 at r6:
nit: mention that IS NOT NULL is only used when IS DISTINCT FROM NULL is not a valid comparison.

Fixes cockroachdb#83754 Previously, the COALESCE and NULLIF operators could error out in vectorized execution when comparing a VOID datum with NULL. This occurred due to the unique property of a VOID, that it can't be compared with another VOID using any of the normal operators such as `=`, `<`, `>`..., or even IS [NOT] [DISTINCT FROM], for example: ``` SELECT ''::VOID IS DISTINCT FROM ''::VOID; ERROR: unsupported comparison operator: <void> IS DISTINCT FROM <void> SQLSTATE: 22023 ``` Processing of COALESCE in the columnar execution engine for an expression like `COALESCE(void_col, NULL)` builds an IS DISTINCT FROM operation between the VOID column and NULL here: https://github.com/cockroachdb/cockroach/blob/ea559dfe0ba57259ca71d3c8ca1de6388954ea73/pkg/sql/colexec/colbuilder/execplan.go#L2122-L2129 This comparison is assumed to be OK, but fails with an internal error because comparison with `UnknownType` is only implicitly supported if the type can be compared with itself. To address this, this patch modifies vectorized COALESCE to use the IS NOT NULL operator internally instead of IS DISTINCT FROM whenever the latter is not a valid comparison. A similar problem with NULLIF, which internally uses `=`, is fixed. Type checking of NULLIF is enhanced in the parser so incompatible comparisons can be caught prior to execution time. Release note (bug fix): This patch fixes vectorized evaluation of COALESCE involving expressions of type VOID and enhances type checking of NULLIF expressions with VOID, so incompatible comparisons can be caught during query compilation instead of during query execution.

msirek

TFTRs!
bors r+

Reviewable status: complete! 0 of 0 LGTMs obtained (and 2 stale) (waiting on @otan and @yuzefovich)

-- commits line 27 at r6:

Previously, mgartner (Marcus Gartner) wrote…

nit: mention that IS NOT NULL is only used when IS DISTINCT FROM NULL is not a valid comparison.

Done

craig · 2022-07-18T18:03:08Z

Build failed (retrying...):

GitHub CI (Cockroach)

craig · 2022-07-18T19:28:43Z

Build succeeded:

GitHub CI (Cockroach)

msirek requested review from mgartner, otan and a team July 5, 2022 23:11

msirek requested a review from a team as a code owner July 5, 2022 23:11

yuzefovich reviewed Jul 5, 2022

View reviewed changes

otan reviewed Jul 5, 2022

View reviewed changes

msirek commented Jul 5, 2022

View reviewed changes

msirek commented Jul 6, 2022

View reviewed changes

msirek force-pushed the 83754 branch from 2a4526b to a05eaa5 Compare July 6, 2022 00:03

blathers-crl bot requested a review from otan July 6, 2022 00:03

msirek commented Jul 6, 2022

View reviewed changes

otan approved these changes Jul 6, 2022

View reviewed changes

msirek force-pushed the 83754 branch from a05eaa5 to 8879e16 Compare July 6, 2022 15:33

msirek commented Jul 6, 2022

View reviewed changes

msirek commented Jul 8, 2022

View reviewed changes

msirek changed the title ~~tree: fix missing IsDistinctFrom support for VOID compared with NULL~~ tree: fix vectorized COALESCE and NULLIF type checking with VOID Jul 12, 2022

msirek force-pushed the 83754 branch from 8879e16 to 356d0ef Compare July 12, 2022 03:49

msirek commented Jul 12, 2022

View reviewed changes

yuzefovich approved these changes Jul 12, 2022

View reviewed changes

msirek force-pushed the 83754 branch 2 times, most recently from 1054063 to 4fb8726 Compare July 12, 2022 15:07

msirek force-pushed the 83754 branch from 4fb8726 to 4b7372f Compare July 12, 2022 21:57

msirek commented Jul 12, 2022

View reviewed changes

mgartner approved these changes Jul 18, 2022

View reviewed changes

msirek force-pushed the 83754 branch from 4b7372f to 3cc04aa Compare July 18, 2022 15:56

msirek commented Jul 18, 2022

View reviewed changes

craig bot merged commit b29f48b into cockroachdb:master Jul 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tree: fix vectorized COALESCE and NULLIF type checking with VOID #83868

tree: fix vectorized COALESCE and NULLIF type checking with VOID #83868

msirek commented Jul 5, 2022 •

edited

Loading

cockroach-teamcity commented Jul 5, 2022

yuzefovich left a comment

otan Jul 5, 2022

msirek left a comment

otan commented Jul 5, 2022

otan commented Jul 5, 2022

msirek left a comment

msirek left a comment

otan commented Jul 6, 2022

otan commented Jul 6, 2022

msirek left a comment

otan left a comment

msirek left a comment

msirek left a comment

craig bot commented Jul 8, 2022

msirek commented Jul 8, 2022

craig bot commented Jul 9, 2022

msirek commented Jul 9, 2022

craig bot commented Jul 9, 2022

mgartner commented Jul 11, 2022

msirek left a comment

yuzefovich left a comment

mgartner commented Jul 12, 2022

mgartner commented Jul 12, 2022

msirek left a comment

rytaft commented Jul 18, 2022

mgartner left a comment

msirek left a comment

craig bot commented Jul 18, 2022

craig bot commented Jul 18, 2022

	whens[i] = &tree.When{
	Cond: tree.NewTypedComparisonExpr(
	treecmp.MakeComparisonOperator(treecmp.IsDistinctFrom),
	t.Exprs[i].(tree.TypedExpr),
	tree.DNull,
	),
	Val: t.Exprs[i],
	}

	func (e evaluator) EvalCoalesceExpr(expr tree.CoalesceExpr) (tree.Datum, error) {
	for _, ex := range expr.Exprs {
	d, err := ex.(tree.TypedExpr).Eval(e)
	if err != nil {
	return nil, err
	}
	if d != tree.DNull {
	return d, nil

tree: fix vectorized COALESCE and NULLIF type checking with VOID #83868

tree: fix vectorized COALESCE and NULLIF type checking with VOID #83868

Conversation

msirek commented Jul 5, 2022 • edited Loading

cockroach-teamcity commented Jul 5, 2022

yuzefovich left a comment

Choose a reason for hiding this comment

otan Jul 5, 2022

Choose a reason for hiding this comment

msirek left a comment

Choose a reason for hiding this comment

otan commented Jul 5, 2022

otan commented Jul 5, 2022

msirek left a comment

Choose a reason for hiding this comment

msirek left a comment

Choose a reason for hiding this comment

otan commented Jul 6, 2022

otan commented Jul 6, 2022

msirek left a comment

Choose a reason for hiding this comment

otan left a comment

Choose a reason for hiding this comment

msirek left a comment

Choose a reason for hiding this comment

msirek left a comment

Choose a reason for hiding this comment

craig bot commented Jul 8, 2022

msirek commented Jul 8, 2022

craig bot commented Jul 9, 2022

msirek commented Jul 9, 2022

craig bot commented Jul 9, 2022

mgartner commented Jul 11, 2022

msirek left a comment

Choose a reason for hiding this comment

yuzefovich left a comment

Choose a reason for hiding this comment

mgartner commented Jul 12, 2022

mgartner commented Jul 12, 2022

msirek left a comment

Choose a reason for hiding this comment

rytaft commented Jul 18, 2022

mgartner left a comment

Choose a reason for hiding this comment

msirek left a comment

Choose a reason for hiding this comment

craig bot commented Jul 18, 2022

craig bot commented Jul 18, 2022

msirek commented Jul 5, 2022 •

edited

Loading