Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source MongoDB - Added milliseconds to convert timestamp to datetime format #8046

Merged
merged 4 commits into from
Nov 18, 2021
Merged

🐛 Source MongoDB - Added milliseconds to convert timestamp to datetime format #8046

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8c52b25
Added milliseconds to convert timestamp to datetime format
andriikorotkov Nov 17, 2021
9df0130
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 18, 2021
9e2895b
updated docs
andriikorotkov Nov 18, 2021
4756db6
updated docs
andriikorotkov Nov 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ain/resources/config/STANDARD_SOURCE_DEFINITION/b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"sourceDefinitionId": "b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e",
"name": "MongoDb",
"dockerRepository": "airbyte/source-mongodb-v2",
"dockerImageTag": "0.1.3",
"dockerImageTag": "0.1.5",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/mongodb-v2",
"icon": "mongodb.svg"
}
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@
- name: MongoDb
sourceDefinitionId: b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e
dockerRepository: airbyte/source-mongodb-v2
dockerImageTag: 0.1.3
dockerImageTag: 0.1.5
documentationUrl: https://docs.airbyte.io/integrations/sources/mongodb-v2
icon: mongodb.svg
sourceType: database
Expand Down
7 changes: 7 additions & 0 deletions airbyte-db/lib/src/main/java/io/airbyte/db/DataTypeUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ public class DataTypeUtils {
public static final String DATE_FORMAT_PATTERN = "yyyy-MM-dd'T'HH:mm:ss'Z'";
public static final DateFormat DATE_FORMAT = new SimpleDateFormat(DATE_FORMAT_PATTERN); // Quoted "Z" to indicate UTC, no timezone offset

public static final String DATE_FORMAT_WITH_MILLISECONDS_PATTERN = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'";
public static final DateFormat DATE_FORMAT_WITH_MILLISECONDS = new SimpleDateFormat(DATE_FORMAT_WITH_MILLISECONDS_PATTERN);

public static <T> T returnNullIfInvalid(final DataTypeSupplier<T> valueProducer) {
return returnNullIfInvalid(valueProducer, ignored -> true);
}
Expand All @@ -37,6 +40,10 @@ public static <T> T returnNullIfInvalid(final DataTypeSupplier<T> valueProducer,
}
}

public static String toISO8601StringWithMilliseconds(final long epochMillis) {
return DATE_FORMAT_WITH_MILLISECONDS.format(Date.from(Instant.ofEpochMilli(epochMillis)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from task definition: The cursor should capture the exact data with microseconds
Not milliseconds

}

public static String toISO8601String(final long epochMillis) {
return DATE_FORMAT.format(Date.from(Instant.ofEpochMilli(epochMillis)));
}
Expand Down
2 changes: 1 addition & 1 deletion airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ private static ObjectNode readField(final BsonReader reader,
case INT64 -> o.put(fieldName, reader.readInt64());
case DOUBLE -> o.put(fieldName, reader.readDouble());
case DECIMAL128 -> o.put(fieldName, toDouble(reader.readDecimal128()));
case TIMESTAMP -> o.put(fieldName, DataTypeUtils.toISO8601String(reader.readTimestamp().getValue()));
case TIMESTAMP -> o.put(fieldName, DataTypeUtils.toISO8601StringWithMilliseconds(reader.readTimestamp().getValue()));
case DATE_TIME -> o.put(fieldName, DataTypeUtils.toISO8601String(reader.readDateTime()));
case BINARY -> o.put(fieldName, toByteArray(reader.readBinaryData()));
case SYMBOL -> o.put(fieldName, reader.readSymbol());
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-mongodb-v2/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.1.3
LABEL io.airbyte.version=0.1.5
LABEL io.airbyte.name=airbyte/source-mongodb-v2
2 changes: 1 addition & 1 deletion .../airbyte/integrations/io/airbyte/integration_tests/sources/MongoDbSourceDataTypeTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ private List<AirbyteMessage> getExpectedMessages() {
.put("int64", 9223372036854775807L)
.put("double", 1.7976931348623157E308)
.put("decimal", NaN)
.put("tms", DataTypeUtils.toISO8601String(MILLI))
.put("tms", DataTypeUtils.toISO8601StringWithMilliseconds(MILLI))
.put("dateTime", DataTypeUtils.toISO8601String(MILLI))
.put("binary", new BsonBinary(new UUID(10, 15)).getData())
.put("symbol", "s")
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/mongodb-v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ For more information regarding configuration parameters, please see [MongoDb Doc

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.5 | 2021-11-17 | [7160](https://github.com/airbytehq/airbyte/pull/7160) | Added milliseconds to convert timestamp to datetime format |
| 0.1.3 | 2021-10-19 | [7160](https://github.com/airbytehq/airbyte/pull/7160) | Fixed nested document parsing |
| 0.1.2 | 2021-10-07 | [6860](https://github.com/airbytehq/airbyte/pull/6860) | Added filter to avoid MongoDb system collections |
| 0.1.1 | 2021-09-21 | [6364](https://github.com/airbytehq/airbyte/pull/6364) | Source MongoDb: added support via TLS/SSL |
Expand Down