Should the default number type in numberscope formulas be bigint or rational?

[gwhitney]

mathjs now supports both bigints, and exact rational numbers (represented as a pair of coprime bigints). It is not entirely clear to me which one we want to configure it to use as the default number type. As just one of many many examples, if the user types n^2 / 8 + 3 /4 (presuming that to get the integer sequence there will be a floor operation at the end), are they expecting the value 1 when n=2 (since 4/8 + 3/4 is bigger than 1) or 0 (because 4/8 is 0 in bigint arithmetic, and 3/4 is 0 in bigint arithmetic, and 0+0 = 0)? In this case, the "+ 3/4" is telltale: in bigint arithmetic, that's a useless "+ 0", so they must have meant rational arithmetic. I don't think that we can reasonably expect to decipher these clues and pick on a formula-by-formula basis which one to use as the number type of the formula.

Therefore, I am leaning toward simply using Fraction (the exact rational type) as the default number type in our formulas. It always seems possible to recover what the formula-writer meant, except if they intentionally wanted / to mean the integer part of the quotient. But I think if we don't give Numberscope visitors any reason to think that / is going to truncate quotients to the integer part, and do give them an easy way to write floor(a/b), then that's not a big worry.

As to this latter, I do intend to support |_blah_| as an alternate notation (from AsciiMath) for floor(blah), as it seems more comfortable; and in the case of division in particular we could also follow python and support a // b to mean floor(a/b). Then if we advertise these, it should also help to avoid any illusions that a/b might do the flooring automatically.

There are some quirks in the implementation of the two types in mathjs. For example, sqrt(bigint(2)) silently converts the bigint to a built-in number and so gives a 53-bit approximation to the square root, whereas sqrt(fraction(2)) fails with an error. But I don't think this should be the deciding factor for us; I think we should instead iron out those quirks in mathjs. For example, mathjs has a "precision" configuration parameter on a log-base-10 scale, for calculations with its "arbitrary-precision decimal" type; we could just arrange that functions with irrational results return a rational value r for which the correct answer is in the range r ± 10^-precision. In the case of sqrt, for example, it would presumably use a continued fraction until the error bound ended up in this range. Other functions like exp and log will be trickier, but these are all solved problems, and the 'BigNumber' decimal class does have explicit guarantees on its results that can be adapted.

[gwhitney]

Reflecting on this question, it may not quite be framed correctly. Maybe there is not a particular "default number type". We know the OEIS sequence inputs to formulas will be bigints. So presumably we want to interpret each integer appearing in a formula as a bigint. And then we want any ratio of bigints to be an exact rational ("Fraction" in the mathjs type system), unless it is explicitly truncated as in floor(a/b) (or |_a/b_| or a // b if we decide to adopt either of those notations). All those things seem uncontroversial to me. So the nuggets of this question that remain seem to be:

1. How should we interpret a decimal value that someone types into a formula, like 1.62, say. As an exact rational 81/50? As a floating point number? If the latter, how much precision do we want to use in the floating-point representation? The options are 53 bits of mantissa, in the JavaScript built-in number type (IEEE double-precision arithmetic), or a configurable number of decimal digits of mantissa (!, yes, it is actually doing decimal arithmetic for some unfathomable reason), using the BigNumber type. Then presumably any operations between a floating-point quantity and an exact quantity are converted to floating-point.

2. When an exact integer or rational enters an operation with an irrational outcome, how do we want to represent the result? As the rational with lowest denominator within some precision (that we set) of the correct answer? Or convert to floating point, and then again if so, what representation?

3. In any of these scenarios, when we do enter approximated values but then encounter an operation like floor that purports to produce an exact result, do we in the long run want to be able to track the error precisely and know if we can guarantee the correctness of the outcome? If yes, then ultimately we will likely have to switch to some sort of interval or "ball" representation of approximate real numbers (i.e. in some way track closed intervals that we know the true value lies in, and then if the outcome is an interval on which the result is ambiguous, attempt to re-do the computation with smaller intervals to remove the ambiguity, and eventually give up with an error or warning).

[gwhitney]

In the color computation PR #530 we are going to implement a scheme in which each formula can have a preferred type, which the infrastructure tries to convert to for you. That would seem to supersede this discussion, so I think it can be closed once that PR is merged.