Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): use nanosecond precision by default in Point.toLineProtocol() #338

Merged
merged 9 commits into from
May 16, 2021
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
## 1.14.0 [unreleased]

### Features

1. [#338](https://github.com/influxdata/influxdb-client-js/pull/338): Use nanosecond precision by default in `Point.toLineProtocol()`.

## 1.13.0 [2021-04-30]

### Features
Expand Down
21 changes: 15 additions & 6 deletions packages/core/src/Point.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
import {convertTimeToNanos} from './util/currentTime'
import {escape} from './util/escape'

/**
* Settings that control the way of how a {@link Point} is serialized
* to a protocol line.
*/
export interface PointSettings {
/** default tags to add to every point */
defaultTags?: {[key: string]: string}
/** convertTime serializes Point's timestamp to a line protocol value */
convertTime?: (
value: string | number | Date | undefined
) => string | undefined
Expand Down Expand Up @@ -126,10 +129,13 @@ export class Point {
* Sets point timestamp. Timestamp can be specified as a Date (preferred), number, string
* or an undefined value. An undefined value instructs to assign a local timestamp using
* the client's clock. An empty string can be used to let the server assign
* the timestamp. A number value represents time as a count of time units since epoch.
* The current time in nanoseconds can't precisely fit into a JS number, which
* can hold at most 2^53 integer number. Nanosecond precision numbers are thus supplied as
* a (base-10) string. An application can use ES2020 BigInt to represent nanoseconds,
* the timestamp. A number value represents time as a count of time units since epoch, the
* exact time unit then depends on the {@link InfluxDB.getWriteApi | precision} of the API
* that writes the point.
*
* Beware that the current time in nanoseconds can't precisely fit into a JS number,
* which can hold at most 2^53 integer number. Nanosecond precision numbers are thus supplied as
* a (base-10) string. An application can also use ES2020 BigInt to represent nanoseconds,
* BigInt's `toString()` returns the required high-precision string.
*
* Note that InfluxDB requires the timestamp to fit into int64 data type.
Expand All @@ -144,10 +150,11 @@ export class Point {

/**
* Creates an InfluxDB protocol line out of this instance.
* @param settings - settings define the exact representation of point time and can also add default tags
* @param settings - settings control serialization of a point timestamp and can also add default tags,
* nanosecond timestamp precision is used when no `settings` or `settings.convertTime` is supplied.
* @returns an InfluxDB protocol line out of this instance
*/
public toLineProtocol(settings?: PointSettings): string | undefined {
public toLineProtocol(settings?: Partial<PointSettings>): string | undefined {
if (!this.name) return undefined
let fieldsLine = ''
Object.keys(this.fields)
Expand Down Expand Up @@ -179,6 +186,8 @@ export class Point {
let time = this.time
if (settings && settings.convertTime) {
time = settings.convertTime(time)
} else {
time = convertTimeToNanos(time)
}

return `${escape.measurement(this.name)}${tagsLine} ${fieldsLine}${
Expand Down
21 changes: 21 additions & 0 deletions packages/core/src/util/currentTime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,24 @@ export const dateToProtocolTimestamp = {
us: (d: Date): string => `${d.getTime()}000`,
ns: (d: Date): string => `${d.getTime()}000000`,
}

/**
* convertTimeToNanos converts of Point's timestamp to a string
* @param value - supported timestamp value
* @returns line protocol value
*/
export function convertTimeToNanos(
value: string | number | Date | undefined
): string | undefined {
if (value === undefined) {
return nanos()
} else if (typeof value === 'string') {
return value.length > 0 ? value : undefined
} else if (value instanceof Date) {
return `${value.getTime()}000000`
} else if (typeof value === 'number') {
return String(Math.floor(value))
Copy link

@empz empz May 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused by this line. The fact that the timestamp value is a number doesn't mean it's in NS, or does it?

Shouldn't this block be doing the following?

return Math.floor(value).toString().padEnd(19, '0')

This way it would ensure that no matter if the Point was constructed given a number in seconds or milliseconds, it'll allways pad it with 0s until reaching NS precision.

Copy link
Contributor Author

@sranka sranka May 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing a number never implied conversion to nanoseconds (even by default). A Date instance should be better used, it was/is a preferred choice since it contains an exact timestamp, it does not need further explanation. If you pass a number, its time unit (precision) is external to a point + you also cannot express precise nanoseconds with a number type. This all was the API contract before this PR, it was also documented that way.

Padding a number to 19 digits is not compliant with the previous behavior, it is also not something that I would like to introduce. Users can still use a string or a Date value for what they need, but a number value already has a different meaning to what you expect. Maybe that I am too deep in the js client, so that the documentation that seems clear to me is not clear enough :) I would appreciate your suggestions about changing the doc.

https://influxdata.github.io/influxdb-client-js/influxdb-client.point.timestamp.html :

Point.timestamp() method
Sets point timestamp. Timestamp can be specified as a Date (preferred), number, string or an undefined value. An
undefined value instructs to assign a local timestamp using the client’s clock. An empty string can be used to let the server
assign the timestamp. A number value represents time as a count of time units since epoch. The current time in
nanoseconds can’t precisely fit into a JS number, which can hold at most 2^53 integer number. Nanosecond precision
numbers are thus supplied as a (base-10) string. An application can use ES2020 BigInt to represent nanoseconds,
BigInt’s toString() returns the required high-precision string.

Note that InfluxDB requires the timestamp to fit into int64 data type.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A number value represents time as a count of time units since epoch

That means I can do Point.timestamp(1620908660) (seconds) but also Point.timestamp(1620908660999) (milliseconds) ?

Your convertTimeToNanos function is supposed to return the timestamp in nanoseconds as a string, right?

So how does the following work?

const point = new Point("measurement").timestamp(1620908660); // seconds

console.log(point.toLineProtocol());

That falls into the block

else if (typeof value === 'number') {
    return String(Math.floor(value))
}

So the timestamp is simply converted to a string without adding the extra characters needed to be in nanosecond precision.

Or what am I missing here?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...but a number value already has a different meaning to what you expect this is the part that is not clear, or at least it's not what I would expect.

If I say point.toLineProtocol() I expect to get the same output format no matter how the point was built. If this is not the case and making it so is not an option because it's a breaking change, I think the documentation for toLineProtocol() should make this very clear.

Copy link
Contributor Author

@sranka sranka May 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A number does not imply a nanosecond precision, it is a fact to accept independently what I or you think about. I am welcome to a documentation change for you to suggest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I improved the doc in the Point class in the scope of this PR, I think that it is more explanatory now.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/**
   * Creates an InfluxDB protocol line out of this instance.
   * @param settings - settings control serialization of a point timestamp and can also add default tags,
   * nanosecond timestamp precision is used when no settings are supplied.
   * @returns an InfluxDB protocol line out of this instance
   */
  public toLineProtocol(settings?: Partial<PointSettings>): string | undefined { }

I'm sorry to be a pain but I don't think this is an improvement at all. The added doc. comment says "nanosecond timestamp precision is used when no settings are supplied" which is not true.

Again, if I create a point like this:

const point = new Point("measurement").timestamp(1620908660); // seconds

Then do point.toLineProtocol(), without settings. The new doc states that I'll get nanosecond precision, which I won't because of this block:

else if (typeof value === 'number') {
    return String(Math.floor(value))
}

But either I'm crazy or this is really confusing. It would be nice to have a third opinion... @vlastahajek ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the confusion comes from the fact that toLineProtocol is a public method and you're implying it will always be used with the writeApi from this library.
My use case was different, I was simply creating points, serializing them with toLineProtocol() and sent them to a message broker for Telegraf to consume and write to InfluxDB 2.0, which failed because the timestamp wasn't in nanoseconds.
So it's kind of an inconsistency between this library, Telegraf and its influxdb_v2 output plugin.

Having said this, I still think the "toLineProtocol" method should use the same precision in the output no matter how the point was built in the first place.

For now, I'll just create my points with nanosecond strings for my use case.

Copy link
Contributor Author

@sranka sranka May 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that telegraf requires nanos and I know your UC since the beginning, I was quite recently sending line protocol data to telegraf through kafka. A developer that uses a number timestamp has to follow the improved documentation of Point.timestamp(). point.toLineProtocol() indeed uses nanosecond precision without settings, it really means that the supplied number is in nanoseconds. For example: 999999999999 number is

  • 1970-01-01T00:16:39.999999999Z with nanosecond precision
  • 2001-09-09T01:46:39.999Z with millisecond precision

The Point implementation is correct in the expectation now, 999999999999 cannot be 2286-11-20T17:46:39.999Z as you suggest. A number is quite problematic to absorb, that's why I already recommended to use Date instances.

} else {
return String(value)
}
}
42 changes: 41 additions & 1 deletion packages/core/test/unit/Point.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ function createPoint(test: PointTest): Point {
})
if (test.time) {
point.timestamp(test.time)
} else {
point.timestamp('')
}
return point
}
Expand Down Expand Up @@ -98,10 +100,48 @@ describe('Point', () => {
it('creates line with JSON double encoded field #241', () => {
const fieldValue = JSON.stringify({prop: JSON.stringify({str: 'test'})})
const point = new Point('tst')
point.stringField('a', fieldValue)
point.stringField('a', fieldValue).timestamp('')
expect(point.toLineProtocol()).equals(
'tst a="{X"propX":X"{XXX"strXXX":XXX"testXXX"}X"}"'.replace(/X/g, '\\')
)
})
it('serializes Point with current time nanosecond OOTB', () => {
const point = new Point('tst').floatField('a', 1)
const lpParts = point.toLineProtocol()?.split(' ') as string[]
expect(lpParts).has.length(3)
expect(lpParts[0]).equals('tst')
expect(lpParts[1]).equals('a=1')
// expect current time in nanoseconds
const nowMillisStr = String(Date.now())
expect(lpParts[2]).has.length(nowMillisStr.length + 6)
expect(
Number.parseInt(lpParts[2].substring(0, nowMillisStr.length)) - 1
).lte(Date.now())
})
it("serializes Point's Date timestamp with nanosecond precision OOTB", () => {
const point = new Point('tst').floatField('a', 1).timestamp(new Date())
const lpParts = point.toLineProtocol()?.split(' ') as string[]
expect(lpParts).has.length(3)
expect(lpParts[0]).equals('tst')
expect(lpParts[1]).equals('a=1')
// expect current time in nanoseconds
const nowMillisStr = String(Date.now())
expect(lpParts[2]).has.length(nowMillisStr.length + 6)
expect(
Number.parseInt(lpParts[2].substring(0, nowMillisStr.length)) - 1
).lte(Date.now())
})
it("serializes Point's number timestamp as-is OOTB", () => {
const point = new Point('tst').floatField('a', 1).timestamp(1)
expect(point.toLineProtocol()).equals('tst a=1 1')
})
it("serializes Point's unknown timestamp as-is OOTB", () => {
const point = new Point('tst').floatField('a', 1).timestamp(({
toString() {
return 'any'
},
} as unknown) as undefined)
expect(point.toLineProtocol()).equals('tst a=1 any')
})
})
})