Tap refactoring using singer generator #47
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RushiT0122
reviewed
Mar 6, 2025
Comment on lines
+70
to
+77
| ```json | ||
| { | ||
| "start_date": "2019-01-01T00:00:00Z", | ||
| "user_agent": "tap-appsflyer <api_user_email@your_company.com>", | ||
| "app_id": "abc1e2swewe", | ||
| "api_token": "askawqewdqwer123445666" | ||
| ... | ||
| } |
There was a problem hiding this comment.
Suggested change
| ```json | |
| { | |
| "start_date": "2019-01-01T00:00:00Z", | |
| "user_agent": "tap-appsflyer <api_user_email@your_company.com>", | |
| "app_id": "abc1e2swewe", | |
| "api_token": "askawqewdqwer123445666" | |
| ... | |
| } | |
| ```json | |
| { | |
| "start_date": "2019-01-01T00:00:00Z", | |
| "user_agent": "tap-appsflyer <api_user_email@your_company.com>", | |
| "app_id": "abc1e2swewe", | |
| "api_token": "askawqewdqwer123445666" | |
| ... | |
| }``` |
RushiT0122
reviewed
Mar 6, 2025
Comment on lines
+95
to
+98
| See the Singer docs on discovery mode | ||
| [here](https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md | ||
|
|
||
| 5. Run the Tap in Sync Mode (with catalog) and [write out to state file](https://github.com/singer-io/getting-started/blob/master/docs/RUNNING_AND_DEVELOPING.md |
There was a problem hiding this comment.
Suggested change
| See the Singer docs on discovery mode | |
| [here](https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md | |
| 5. Run the Tap in Sync Mode (with catalog) and [write out to state file](https://github.com/singer-io/getting-started/blob/master/docs/RUNNING_AND_DEVELOPING.md | |
| See the Singer docs on discovery mode | |
| [here](https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md) | |
| 5. Run the Tap in Sync Mode (with catalog) and [write out to state file](https://github.com/singer-io/getting-started/blob/master/docs/RUNNING_AND_DEVELOPING.md) |
RushiT0122
reviewed
Mar 6, 2025
|
|
||
| To [check the tap](https://github.com/singer-io/singer-tools) | ||
| ```bash | ||
| > tap-mixpanel --config tap_config.json --catalog catalog.json | singer-check-tap > state.json |
There was a problem hiding this comment.
Suggested change
| > tap-mixpanel --config tap_config.json --catalog catalog.json | singer-check-tap > state.json | |
| > tap-appsflyer --config tap_config.json --catalog catalog.json | singer-check-tap > state.json |
RushiT0122
reviewed
Mar 6, 2025
| data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md). | ||
| This is a [Singer](https://singer.io) tap that produces JSON-formatted data | ||
| following the [Singer | ||
| spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md). |
There was a problem hiding this comment.
Suggested change
| spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md). | |
| spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md). |
RushiT0122
reviewed
Mar 6, 2025
| url="http://singer.io", | ||
| classifiers=["Programming Language :: Python :: 3 :: Only"], | ||
| py_modules=["tap_appsflyer"], | ||
| install_requires= ["attrs==25.1.0", "singer-python==6.1.0", "requests==2.32.3", "backoff==2.2.1"], |
There was a problem hiding this comment.
Suggested change
| install_requires= ["attrs==25.1.0", "singer-python==6.1.0", "requests==2.32.3", "backoff==2.2.1"], | |
| install_requires= ["attrs==25.1.0", | |
| "singer-python==6.1.0", | |
| "requests==2.32.3", | |
| "backoff==2.2.1"], |
RushiT0122
reviewed
Mar 6, 2025
Comment on lines
+32
to
+35
| sync(client=client, | ||
| config=parsed_args.config, | ||
| catalog=parsed_args.catalog, | ||
| state=state) |
There was a problem hiding this comment.
Suggested change
| sync(client=client, | |
| config=parsed_args.config, | |
| catalog=parsed_args.catalog, | |
| state=state) | |
| sync(client=client, | |
| config=parsed_args.config, | |
| catalog=parsed_args.catalog, | |
| state=state) |
RushiT0122
reviewed
Mar 6, 2025
Comment on lines
+54
to
+57
| if config_request_timeout and float(config_request_timeout): | ||
| self.request_timeout = float(config_request_timeout) | ||
| else: | ||
| self.request_timeout = REQUEST_TIMEOUT |
There was a problem hiding this comment.
Logically your code is correct, just a suggestion.
Suggested change
| if config_request_timeout and float(config_request_timeout): | |
| self.request_timeout = float(config_request_timeout) | |
| else: | |
| self.request_timeout = REQUEST_TIMEOUT | |
| self.request_timeout = float(config_request_timeout) if config_request_timeout else REQUEST_TIMEOUT |
RushiT0122
reviewed
Mar 7, 2025
Comment on lines
+206
to
+217
| while True: | ||
| response = self.client.get( | ||
| extraction_url, self.params, self.headers | ||
| ) | ||
| if not response: | ||
| LOGGER.warning("No records found on Page %s", page_count) | ||
| break | ||
|
|
||
| with singer.metrics.http_request_timer(self.parse_source_from_url(self.client.base_url)) as timer: | ||
| resp = SESSION.send(response) | ||
| timer.tags[singer.metrics.Tag.http_status_code] = resp.status_code | ||
| return resp |
RushiT0122
reviewed
Mar 7, 2025
Comment on lines
+290
to
+293
| if value.lower() == "TRUE".lower(): | ||
| record[field_name] = True | ||
| else: | ||
| record[field_name] = False |
There was a problem hiding this comment.
Suggested change
| if value.lower() == "TRUE".lower(): | |
| record[field_name] = True | |
| else: | |
| record[field_name] = False | |
| record[field_name] = value.lower() == "true" |
RushiT0122
reviewed
Mar 7, 2025
Comment on lines
+312
to
+315
| from_datetime = bookmark_date =self.get_bookmark(state) | ||
| to_datetime = self.get_stop(from_datetime, datetime.datetime.now(pytz.utc)) | ||
|
|
||
| current_max_bookmark_date = bookmark_date_to_utc = bookmark_date |
There was a problem hiding this comment.
Suggested change
| from_datetime = bookmark_date =self.get_bookmark(state) | |
| to_datetime = self.get_stop(from_datetime, datetime.datetime.now(pytz.utc)) | |
| current_max_bookmark_date = bookmark_date_to_utc = bookmark_date | |
| from_datetime = bookmark_date | |
| bookmark_date = self.get_bookmark(state) | |
| to_datetime = self.get_stop(from_datetime, datetime.datetime.now(pytz.utc)) | |
| current_max_bookmark_date = bookmark_date_to_utc | |
| bookmark_date_to_utc = bookmark_date |
There was a problem hiding this comment.
Do the same wherever required?
RushiT0122
reviewed
Mar 7, 2025
Comment on lines
+324
to
+337
| try: | ||
| first_line = next(csv_data) | ||
| # Push the line back into the iterator | ||
| csv_data = iter([first_line] + list(csv_data)) | ||
| except StopIteration: | ||
| LOGGER.warning("No data available in the CSV.") | ||
| return state | ||
| reader = csv.DictReader(csv_data, fieldnames) | ||
|
|
||
| try: | ||
| next(reader) # Skip the header row | ||
| except StopIteration: | ||
| LOGGER.warning("No data available after header row.") | ||
| return state |
There was a problem hiding this comment.
Suggested change
| try: | |
| first_line = next(csv_data) | |
| # Push the line back into the iterator | |
| csv_data = iter([first_line] + list(csv_data)) | |
| except StopIteration: | |
| LOGGER.warning("No data available in the CSV.") | |
| return state | |
| reader = csv.DictReader(csv_data, fieldnames) | |
| try: | |
| next(reader) # Skip the header row | |
| except StopIteration: | |
| LOGGER.warning("No data available after header row.") | |
| return state | |
| try: | |
| reader = csv.DictReader(csv_data, fieldnames) | |
| next(reader) # Skip the header row | |
| except StopIteration: | |
| LOGGER.warning("No data available in the CSV.") | |
| return state |
RushiT0122
reviewed
Mar 7, 2025
Comment on lines
+339
to
+354
| for i, row in enumerate(reader): | ||
| xform_record = self.xform(row) | ||
| transformed_record = transformer.transform( | ||
| xform_record, schema, stream_metadata | ||
| ) | ||
| try: | ||
| record_timestamp = strptime_to_utc( | ||
| transformed_record[self.replication_keys[0]] | ||
| ) | ||
| except KeyError as _: | ||
| LOGGER.error( | ||
| "Unable to process Record, Exception occurred: %s for stream %s", | ||
| _, | ||
| self.__class__, | ||
| ) | ||
| continue |
There was a problem hiding this comment.
Suggested change
| for i, row in enumerate(reader): | |
| xform_record = self.xform(row) | |
| transformed_record = transformer.transform( | |
| xform_record, schema, stream_metadata | |
| ) | |
| try: | |
| record_timestamp = strptime_to_utc( | |
| transformed_record[self.replication_keys[0]] | |
| ) | |
| except KeyError as _: | |
| LOGGER.error( | |
| "Unable to process Record, Exception occurred: %s for stream %s", | |
| _, | |
| self.__class__, | |
| ) | |
| continue | |
| for _, row in enumerate(reader): | |
| xform_record = self.xform(row) | |
| transformed_record = transformer.transform( | |
| xform_record, schema, stream_metadata | |
| ) | |
| try: | |
| record_timestamp = strptime_to_utc( | |
| transformed_record[self.replication_keys[0]] | |
| ) | |
| except KeyError as ex: | |
| LOGGER.error( | |
| "Unable to process Record, Exception occurred: %s for stream %s", | |
| ex, | |
| self.__class__, | |
| ) | |
| continue |
RushiT0122
requested changes
Mar 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
singer-pythonupgrade to 6.1.0 andbackoffto 2.2.1Manual QA steps
Risks
Rollback steps
AI generated code
https://internal.qlik.dev/general/ways-of-working/code-reviews/#guidelines-for-ai-generated-code