Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add round, ceil, floor, truncate to the In-Database Column type #6988

Merged
merged 89 commits into from
Jun 30, 2023
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
bbc5cff
one test
GregoryTravis Jun 7, 2023
e6bed7b
function_name
GregoryTravis Jun 7, 2023
347b46e
expression_spec
GregoryTravis Jun 7, 2023
c78560d
gmt test
GregoryTravis Jun 7, 2023
2fa1846
sl
GregoryTravis Jun 7, 2023
8291ce3
Merge branch 'develop' into wip/gmt/6886-truncate
GregoryTravis Jun 7, 2023
b16d9ee
trunc tests
GregoryTravis Jun 8, 2023
be81b60
ceil floor
GregoryTravis Jun 8, 2023
3b486b9
by hand rounding, sqlite parser stack overflow
GregoryTravis Jun 8, 2023
9fe0841
wip
GregoryTravis Jun 8, 2023
f649b73
pg maybe smaller
GregoryTravis Jun 8, 2023
2f68d35
works in sqlite too
GregoryTravis Jun 8, 2023
7f7324b
col math works
GregoryTravis Jun 9, 2023
9c4e0ff
clean up
GregoryTravis Jun 12, 2023
abaab2a
tests, cast to int column
GregoryTravis Jun 12, 2023
e20161a
rounding tests
GregoryTravis Jun 12, 2023
f42285e
round expression spec
GregoryTravis Jun 12, 2023
d802271
int in int out
GregoryTravis Jun 13, 2023
f05f9e3
full col tests
GregoryTravis Jun 13, 2023
ccda6aa
round to float
GregoryTravis Jun 13, 2023
4ff6fa7
dp out of range
GregoryTravis Jun 13, 2023
5b41821
const tests
GregoryTravis Jun 13, 2023
93f4403
rename to const, name test
GregoryTravis Jun 13, 2023
c60d354
nulls
GregoryTravis Jun 13, 2023
a25e3d5
remove pow_flip
GregoryTravis Jun 13, 2023
5283255
make trunc, ceil, floor generic
GregoryTravis Jun 13, 2023
4aed89c
do not restrict const arg
GregoryTravis Jun 14, 2023
50b78eb
Merge branch 'develop' into wip/gmt/6886-truncate
GregoryTravis Jun 14, 2023
fbecfd8
review
GregoryTravis Jun 14, 2023
593982b
nan/inf docs
GregoryTravis Jun 14, 2023
3ff04b5
changelog
GregoryTravis Jun 14, 2023
05fc15e
decimal too
GregoryTravis Jun 14, 2023
3fe8280
restore deleted test, restore integrity check
GregoryTravis Jun 15, 2023
31874a8
const not an op, using make_constant
GregoryTravis Jun 15, 2023
9256930
wip
GregoryTravis Jun 15, 2023
03f3e05
in-mem const
GregoryTravis Jun 15, 2023
78aa29f
share with eval exp
GregoryTravis Jun 15, 2023
92b4d49
factor out dp check
GregoryTravis Jun 15, 2023
8d19564
only test decimal on pg-like
GregoryTravis Jun 15, 2023
91d58b2
column arithmetic for in-mem too
GregoryTravis Jun 15, 2023
ea10375
wip
GregoryTravis Jun 15, 2023
b7b6d3e
symmetric
GregoryTravis Jun 15, 2023
e8dbf9f
symmetric for in-mem
GregoryTravis Jun 16, 2023
218f2e2
symmetric for db
GregoryTravis Jun 16, 2023
570756d
wip
GregoryTravis Jun 16, 2023
afd21ac
integer impl
GregoryTravis Jun 19, 2023
b0c771e
wip
GregoryTravis Jun 19, 2023
bbd7033
col col
GregoryTravis Jun 19, 2023
e891c44
unstable
GregoryTravis Jun 19, 2023
511a627
review
GregoryTravis Jun 19, 2023
9769faf
pg/sl return type
GregoryTravis Jun 20, 2023
41bd417
more type checks for in-mem
GregoryTravis Jun 20, 2023
4d0627b
input check for in-mem round
GregoryTravis Jun 21, 2023
0ad7ef4
use make_function
GregoryTravis Jun 21, 2023
0a350c2
wip
GregoryTravis Jun 21, 2023
d6fe5ce
wrapper
GregoryTravis Jun 22, 2023
395a3b6
round_decimal
GregoryTravis Jun 22, 2023
10fa61c
sl
GregoryTravis Jun 22, 2023
086fb18
round etc return types
GregoryTravis Jun 26, 2023
c6fed25
rename round_with
GregoryTravis Jun 26, 2023
6351dce
cleanup
GregoryTravis Jun 26, 2023
c27b52d
docs
GregoryTravis Jun 26, 2023
314546d
decimal test
GregoryTravis Jun 26, 2023
7fcb5c7
specialize
GregoryTravis Jun 26, 2023
3bac313
remove max_long test
GregoryTravis Jun 26, 2023
05620e4
p flag for dp restriction
GregoryTravis Jun 26, 2023
307f9f0
rs too
GregoryTravis Jun 26, 2023
021a2f4
return type tests
GregoryTravis Jun 26, 2023
6ed5154
bankers
GregoryTravis Jun 26, 2023
519f329
cleanup
GregoryTravis Jun 26, 2023
c6d7974
merge
GregoryTravis Jun 26, 2023
934f974
Merge branch 'develop' into wip/gmt/6886-truncate
GregoryTravis Jun 27, 2023
35fe5b1
literal for dp
GregoryTravis Jun 27, 2023
076172a
changelog chronological
GregoryTravis Jun 27, 2023
3a9b1d7
review
GregoryTravis Jun 27, 2023
819e75f
undo changelog
GregoryTravis Jun 27, 2023
bfce3b7
precision too high
GregoryTravis Jun 27, 2023
e13e178
Merge branch 'develop' into wip/gmt/6886-truncate
GregoryTravis Jun 27, 2023
9d3f548
wip
GregoryTravis Jun 28, 2023
4895366
trunc
GregoryTravis Jun 28, 2023
6342238
DoubleLongMapOpWithSpecialNumericHandling
GregoryTravis Jun 28, 2023
c22364e
wip
GregoryTravis Jun 28, 2023
7b16564
fmt
GregoryTravis Jun 28, 2023
2ca4084
Merge branch 'develop' into wip/gmt/6886-truncate
GregoryTravis Jun 28, 2023
bc6f8fa
precision limit tests
GregoryTravis Jun 29, 2023
9905ac3
fixed
GregoryTravis Jun 29, 2023
74f00d3
cleanup
GregoryTravis Jun 29, 2023
40d2376
epsilon
GregoryTravis Jun 29, 2023
9ff2ef6
another epsilon
GregoryTravis Jun 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1311,7 +1311,8 @@ type Table
Create a constant column from a value.
make_constant_column : Any -> Column
make_constant_column self value =
Column.from_vector_repeated ("Constant_" + UUID.randomUUID.to_text) [value] self.row_count
if Table_Helpers.is_column value then Error.throw (Illegal_Argument.Error "A constant value may only be created from a scalar, not a Column") else
Column.from_vector_repeated ("Constant_" + UUID.randomUUID.to_text) [value] self.row_count

## Returns the vector of columns contained in this table.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package org.enso.table.data.column.operation.map;

import java.util.BitSet;
import org.enso.table.data.column.storage.numeric.DoubleStorage;
import org.enso.table.data.column.storage.numeric.LongStorage;

public abstract class DoubleLongMapOpWithSpecialNumericHandling
extends UnaryMapOperationWithProblemBuilder<Double, DoubleStorage> {
public DoubleLongMapOpWithSpecialNumericHandling(String name) {
super(name);
}

protected abstract long doOperation(double a);

@Override
public LongStorage run(
DoubleStorage storage, Object arg, MapOperationProblemBuilder problemBuilder) {
long[] out = new long[storage.size()];
BitSet isMissing = new BitSet();

for (int i = 0; i < storage.size(); i++) {
if (!storage.isNa(i)) {
double item = storage.getItem(i);
boolean special = Double.isNaN(item) || Double.isInfinite(item);
if (!special) {
out[i] = doOperation(item);
} else {
String msg = "Value is " + item;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error message needs to be more user friendly.

Suggested change
String msg = "Value is " + item;
String msg = "The operation " + name + " does not allow " + item + " as input, returning Nothing.";

or sth like that

problemBuilder.reportArithmeticError(msg, i);
isMissing.set(i);
}
} else {
isMissing.set(i);
}
}
return new LongStorage(out, storage.size(), isMissing);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package org.enso.table.data.column.operation.map;

import org.enso.table.data.column.storage.Storage;

/**
* A map-like operation that ignores its second argument
*
* @param <I> the supported storage type
*/
public abstract class UnaryMapOperationWithProblemBuilder<T, I extends Storage<T>>
extends MapOperation<T, I> {
public UnaryMapOperationWithProblemBuilder(String name) {
super(name);
}

protected abstract Storage<?> run(
I storage, Object arg, MapOperationProblemBuilder problemBuilder);

@Override
public Storage<?> runMap(I storage, Object arg, MapOperationProblemBuilder problemBuilder) {
return run(storage, arg, problemBuilder);
}

@Override
public Storage<?> runZip(I storage, Storage<?> arg, MapOperationProblemBuilder problemBuilder) {
return run(storage, arg, problemBuilder);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
import java.util.List;
import org.enso.table.data.column.builder.Builder;
import org.enso.table.data.column.builder.NumericBuilder;
import org.enso.table.data.column.operation.map.DoubleLongMapOpWithSpecialNumericHandling;
import org.enso.table.data.column.operation.map.MapOpStorage;
import org.enso.table.data.column.operation.map.MapOperationProblemBuilder;
import org.enso.table.data.column.operation.map.UnaryDoubleToLongOp;
import org.enso.table.data.column.operation.map.UnaryMapOperation;
import org.enso.table.data.column.operation.map.numeric.DoubleBooleanOp;
import org.enso.table.data.column.operation.map.numeric.DoubleIsInOp;
Expand Down Expand Up @@ -245,21 +245,21 @@ protected double doDouble(
}
})
.add(
new UnaryDoubleToLongOp(Maps.TRUNCATE) {
new DoubleLongMapOpWithSpecialNumericHandling(Maps.TRUNCATE) {
@Override
protected long doOperation(double a) {
return (long) a;
}
})
.add(
new UnaryDoubleToLongOp(Maps.CEIL) {
new DoubleLongMapOpWithSpecialNumericHandling(Maps.CEIL) {
@Override
protected long doOperation(double a) {
return (long) Math.ceil(a);
}
})
.add(
new UnaryDoubleToLongOp(Maps.FLOOR) {
new DoubleLongMapOpWithSpecialNumericHandling(Maps.FLOOR) {
@Override
protected long doOperation(double a) {
return (long) Math.floor(a);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -512,14 +512,6 @@ spec setup =
table = table_builder [["x", [0, 3, -3, 1, -2]]]
table.at "x" . round 16 . should_fail_with Illegal_Argument

Test.specify "Can handle NaN/Infinity" <|
nan_result = if setup.test_selection.is_nan_and_nothing_distinct then Number.nan else Nothing
ops = [.round, .truncate, .ceil, .floor]
ops.each op->
do_op Number.nan op . should_equal nan_result
do_op Number.positive_infinity op . should_equal Number.positive_infinity
do_op Number.negative_infinity op . should_equal Number.negative_infinity

if setup.test_selection.supports_decimal_type then
Test.specify "should return decimals when rounding decimals" <|
i1 = 9223372036854775807 - 1
Expand Down Expand Up @@ -647,69 +639,55 @@ spec setup =
do_round -3.50001 . should_equal -4
do_round -3.99999 . should_equal -4

Test.specify "Returns the correct type" <|
do_round 231.2 1 . should_be_a Decimal
do_round 231.2 0 . should_be_a Decimal
do_round 231.2 . should_be_a Decimal
do_round 231.2 -1 . should_be_a Decimal

Test.specify "Can round correctly near the precision limit" <|
do_round 1.22222222225 10 . should_equal 1.2222222223
do_round 1.222222222225 11 . should_equal 1.22222222223
do_round 1.2222222222225 12 . should_equal 1.222222222223
do_round 1.22222222222225 13 . should_equal 1.2222222222223
do_round 1.222222222222225 14 . should_equal 1.22222222222223
do_round 1.2222222222222225 15 . should_equal 1.222222222222223

do_round -1.22222222225 10 . should_equal -1.2222222223
do_round -1.222222222225 11 . should_equal -1.22222222223
do_round -1.2222222222225 12 . should_equal -1.222222222223
do_round -1.22222222222225 13 . should_equal -1.2222222222223
do_round -1.222222222222225 14 . should_equal -1.22222222222223
do_round -1.2222222222222225 15 . should_equal -1.222222222222223
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because SQLite fails this test. I could add this one line to the backend-specific test, with different values.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it fail it? Should we be concerned? (if not a big deal, I'm ok with removing it)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to backend-specific tests.


do_round 1.22222222235 10 . should_equal 1.2222222224
do_round 1.222222222235 11 . should_equal 1.22222222224
do_round 1.2222222222235 12 . should_equal 1.222222222224
do_round 1.22222222222235 13 . should_equal 1.2222222222224
do_round 1.222222222222235 14 . should_equal 1.22222222222224
do_round 1.2222222222222235 15 . should_equal 1.222222222222224

do_round -1.22222222235 10 . should_equal -1.2222222224
do_round -1.222222222235 11 . should_equal -1.22222222224
do_round -1.2222222222235 12 . should_equal -1.222222222224
do_round -1.22222222222235 13 . should_equal -1.2222222222224
do_round -1.222222222222235 14 . should_equal -1.22222222222224
do_round -1.2222222222222235 15 . should_equal -1.222222222222224

Test.specify "Can round correctly near the precision limit, using banker's rounding" <|
do_round 1.22222222225 10 use_bankers=True . should_equal 1.2222222222
do_round 1.222222222225 11 use_bankers=True . should_equal 1.22222222222
do_round 1.2222222222225 12 use_bankers=True . should_equal 1.222222222222
do_round 1.22222222222225 13 use_bankers=True . should_equal 1.2222222222222
do_round 1.222222222222225 14 use_bankers=True . should_equal 1.22222222222222
do_round 1.2222222222222225 15 use_bankers=True . should_equal 1.222222222222222

do_round -1.22222222225 10 use_bankers=True . should_equal -1.2222222222
do_round -1.222222222225 11 use_bankers=True . should_equal -1.22222222222
do_round -1.2222222222225 12 use_bankers=True . should_equal -1.222222222222
do_round -1.22222222222225 13 use_bankers=True . should_equal -1.2222222222222
do_round -1.222222222222225 14 use_bankers=True . should_equal -1.22222222222222
do_round -1.2222222222222225 15 use_bankers=True . should_equal -1.222222222222222

do_round 1.22222222235 10 use_bankers=True . should_equal 1.2222222224
do_round 1.222222222235 11 use_bankers=True . should_equal 1.22222222224
do_round 1.2222222222235 12 use_bankers=True . should_equal 1.222222222224
do_round 1.22222222222235 13 use_bankers=True . should_equal 1.2222222222224
do_round 1.222222222222235 14 use_bankers=True . should_equal 1.22222222222224
do_round 1.2222222222222235 15 use_bankers=True . should_equal 1.222222222222224

do_round -1.22222222235 10 use_bankers=True . should_equal -1.2222222224
do_round -1.222222222235 11 use_bankers=True . should_equal -1.22222222224
do_round -1.2222222222235 12 use_bankers=True . should_equal -1.222222222224
do_round -1.22222222222235 13 use_bankers=True . should_equal -1.2222222222224
do_round -1.222222222222235 14 use_bankers=True . should_equal -1.22222222222224
do_round -1.2222222222222235 15 use_bankers=True . should_equal -1.222222222222224

Test.specify "Decimal places out of range" <|
do_round 3.1 16 . should_fail_with Illegal_Argument
Expand Down Expand Up @@ -812,12 +790,6 @@ spec setup =
do_round -12250 -2 use_bankers=True . should_equal -12200
do_round -12251 -2 use_bankers=True . should_equal -12300

Test.specify "Returns the correct type" <|
do_round 231 1 . should_be_a Decimal
do_round 231 0 . should_be_a Decimal
do_round 231 . should_be_a Decimal
do_round 231 -1 . should_be_a Decimal

Test.group prefix+"Text Column Operations" <|
t3 = table_builder [["s1", ["foobar", "bar", "baz", "BAB", Nothing]], ["s2", ["foo", "ar", "a", "b", Nothing]]]
s1 = t3.at "s1"
Expand Down
26 changes: 26 additions & 0 deletions test/Table_Tests/src/Database/Postgres_Spec.enso
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,32 @@ postgres_specific_spec connection db_name setup =
col . cast Value_Type.Integer . truncate . value_type . should_equal Value_Type.Float
col . cast Value_Type.Decimal . truncate . value_type . should_equal Value_Type.Decimal

do_op n op =
table = table_builder [["x", [n]]]
result = table.at "x" |> op
result.to_vector.at 0
do_round n dp=0 use_bankers=False = do_op n (_.round dp use_bankers)

Test.specify "Can handle NaN/Infinity" <|
nan_result = if setup.test_selection.is_nan_and_nothing_distinct then Number.nan else Nothing
ops = [.round, .truncate, .ceil, .floor]
ops.each op->
do_op Number.nan op . should_equal nan_result
do_op Number.positive_infinity op . should_equal Number.positive_infinity
do_op Number.negative_infinity op . should_equal Number.negative_infinity

Test.specify "round returns the correct type" <|
do_round 231.2 1 . should_be_a Decimal
do_round 231.2 0 . should_be_a Decimal
do_round 231.2 . should_be_a Decimal
do_round 231.2 -1 . should_be_a Decimal

Test.specify "round returns the correct type" <|
do_round 231 1 . should_be_a Decimal
do_round 231 0 . should_be_a Decimal
do_round 231 . should_be_a Decimal
do_round 231 -1 . should_be_a Decimal

run_tests connection db_name =
prefix = "[PostgreSQL] "
name_counter = Ref.new 0
Expand Down
26 changes: 26 additions & 0 deletions test/Table_Tests/src/Database/SQLite_Spec.enso
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,32 @@ sqlite_specific_spec prefix connection setup =
col . cast Value_Type.Integer . truncate . value_type . should_equal Value_Type.Integer
col . cast Value_Type.Decimal . truncate . value_type . should_equal Value_Type.Float

do_op n op =
table = table_builder [["x", [n]]]
result = table.at "x" |> op
result.to_vector.at 0
do_round n dp=0 use_bankers=False = do_op n (_.round dp use_bankers)

Test.specify "Can handle NaN/Infinity" <|
nan_result = if setup.test_selection.is_nan_and_nothing_distinct then Number.nan else Nothing
ops = [.round, .truncate, .ceil, .floor]
ops.each op->
do_op Number.nan op . should_equal nan_result
do_op Number.positive_infinity op . should_equal Number.positive_infinity
do_op Number.negative_infinity op . should_equal Number.negative_infinity

Test.specify "round returns the correct type" <|
do_round 231.2 1 . should_be_a Decimal
do_round 231.2 0 . should_be_a Decimal
do_round 231.2 . should_be_a Decimal
do_round 231.2 -1 . should_be_a Decimal

Test.specify "round returns the correct type" <|
do_round 231 1 . should_be_a Decimal
do_round 231 0 . should_be_a Decimal
do_round 231 . should_be_a Decimal
do_round 231 -1 . should_be_a Decimal

sqlite_spec connection prefix =
name_counter = Ref.new 0
table_builder columns =
Expand Down
31 changes: 31 additions & 0 deletions test/Table_Tests/src/In_Memory/Column_Spec.enso
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ from Standard.Base import all

import project.Util

import Standard.Base.Errors.Common.Arithmetic_Error
import Standard.Base.Errors.Common.Index_Out_Of_Bounds
import Standard.Base.Errors.Illegal_Argument.Illegal_Argument
import Standard.Examples
Expand Down Expand Up @@ -169,6 +170,36 @@ spec =
Test.specify "Should error on input of the wrong type" <|
Column.from_vector "foo" ["asdf", "zxcv", "qwer"] . floor . should_fail_with Invalid_Value_Type

Test.group "round/truncate/ceil/floor" <|
do_op n op =
col = Column.from_vector "x" [n]
result = op col
result.to_vector.at 0
do_round n dp=0 use_bankers=False = do_op n (_.round dp use_bankers)

Test.specify "round returns the correct type" <|
do_round 231.2 1 . should_be_a Decimal
do_round 231.2 0 . should_be_a Integer
do_round 231.2 . should_be_a Integer
do_round 231.2 -1 . should_be_a Integer

Test.specify "round returns the correct type" <|
do_round 231 1 . should_be_a Integer
do_round 231 0 . should_be_a Integer
do_round 231 . should_be_a Integer
do_round 231 -1 . should_be_a Integer

Test.specify "nan/inf" <|
ops = [.truncate, .ceil, .floor, .round]
ops.map op->
col = Column.from_vector "x" [2.1, 0.0, Number.nan, Number.positive_infinity, Number.negative_infinity, Nothing, 12.1]
result = op col
result.to_vector . drop 2 . take 4 . should_equal [Nothing, Nothing, Nothing, Nothing]
warnings = Warning.get_all result . map .value
warnings . should_contain <| Arithmetic_Error.Error 'Value is -Infinity (at rows [4]).'
warnings . should_contain <| Arithmetic_Error.Error 'Value is Infinity (at rows [3]).'
warnings . should_contain <| Arithmetic_Error.Error 'Value is NaN (at rows [2]).'

Test.group "Date_Time truncate" <|
Test.specify "should be able to truncate a column of Date_Times" <|
Column.from_vector "foo" [Date_Time.new 2020 10 24 1 2 3, Date_Time.new 2020 10 24 1 2 3] . truncate . should_equal <| Column.from_vector "foo" [Date.new 2020 10 24, Date.new 2020 10 24]
Expand Down