Salesforce make start_date optional and change format to YYYY-MM-DD
#8191
Conversation
github-actions
bot
added
the
area/documentation
augan-rymkhan
requested review from
sergei-solonitcyn,
keu,
eliziario,
vitaliizazmic and
yevhenii-ldv
| @staticmethod | ||
| def format_start_date(start_date): | ||
| """Transform the format `2021-07-25` into the format `2021-07-25T00:00:00Z`""" | ||
| return pendulum.parse(start_date).strftime('%Y-%m-%dT%H:%M:%SZ') |
There was a problem hiding this comment.
What if start_date is None?
There was a problem hiding this comment.
@vitaliizazmic This case is fixed now.
|
/test connector=connectors/source-salesforce
|
| "additionalProperties": false, | ||
| "properties": { | ||
| "client_id": { | ||
| "title": "Client Id", |
There was a problem hiding this comment.
| "title": "Client Id", | |
| "title": "Client ID", |
| "description": "Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don't know what this is, follow this <a href=\"https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b\">guide</a> to retrieve it.", | ||
| "type": "string", | ||
| "airbyte_secret": true | ||
| }, | ||
| "start_date": { | ||
| "description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field uses the \"updated\" field if available, otherwise the \"created\" fields if they are available for a stream.", | ||
| "title": "Start Date", |
There was a problem hiding this comment.
| "title": "Start Date", | |
| "title": "Replication Start Date", |
| "description": "Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don't know what this is, follow this <a href=\"https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b\">guide</a> to retrieve it.", | ||
| "type": "string", | ||
| "airbyte_secret": true | ||
| }, | ||
| "start_date": { | ||
| "description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field uses the \"updated\" field if available, otherwise the \"created\" fields if they are available for a stream.", | ||
| "title": "Start Date", | ||
| "description": "Date in the format 2017-01-25. Any data before this date will not be replicated. This field uses the \"updated\" field if available, otherwise the \"created\" fields if they are available for a stream.", |
There was a problem hiding this comment.
Can you also add a sentence like If not set, then by default all your data is replicated.
| }, | ||
| "is_sandbox": { | ||
| "title": "Is Sandbox?", |
There was a problem hiding this comment.
| "title": "Is Sandbox?", | |
| "title": "Sandbox", |
| }, | ||
| "is_sandbox": { | ||
| "title": "Is Sandbox?", | ||
| "description": "Whether or not the the app is in a Salesforce sandbox. If you do not know what this, assume it is false. We provide more info on this field in the <a href=\"https://docs.airbyte.io/integrations/destinations/salesforce#is_sandbox\">docs</a>.", | ||
| "type": "boolean", | ||
| "default": false | ||
| }, | ||
| "api_type": { |
There was a problem hiding this comment.
can you create an issue to deprecate this option? this is an implementation detail that we should remove eventually
There was a problem hiding this comment.
@sherifnada It's created here
|
/test connector=connectors/source-salesforce
|
|
/publish connector=connectors/source-salesforce
|
|
/publish connector=connectors/source-salesforce
|
augan-rymkhan
deleted the
arymkhan/salesforce-make-start-date-optional
branch
…airbytehq#8191) * Make `start_date` optional * start_date can take format `2021-07-25` * added title to all spec input properties and changed version in Dockerfile * added a new item into salesforce.md changelogs * fixed case when start_date is None * formatted code * changed comments in unit test fixtures * changed spec title fields * updated source and spec yaml Co-authored-by: Auganbay <[email protected]>
What
start_dateoptionaltitleanddescriptionto all properties in Salesforce's input specHow
start_datevariable is used in the query and it is defined as:start_date = next_page_token or stream_date or self.start_dateself.start_dateis from the user input configThe solution provided here: if
start_dateis not None, then addWHERE {self.cursor_field} >= {start_date}clause to the query, otherwise query will fetch all data without filtering by date (also there is a LIMIT clause).format_start_datemethod will transform thestart_datefrom format2021-07-25into the format2021-07-25T00:00:00Z For backward compatibility, it works also withstart_date` which is entered in date time (YYYY-MM-DDTXX:XX:XXZ) format.titlefor all properties inspec.jsonfile.Recommended reading order
x.javay.python🚨 User Impact 🚨
Are there any breaking changes? If yes, please make sure to include it here and in any changelogs with the 🚨🚨 emoji
What is the end result perceived by the user?
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/SUMMARY.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described hereUpdating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described hereConnector Generator
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changes