Low code CDK: With PageIncrement as page strategy, page_token_option is not injected into request parameter

[bala-ceg]

Current Behavior

This is my new connector PR 18893, in which I have used PageIncrement as page strategy and as you can see from the below log in the below outbound API message, the page option is not added in requester

Logs

(.venv) source-aha % python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
{"type": "DEBUG", "message": "Debug logs enabled", "data": {}}
{"type": "LOG", "log": {"level": "INFO", "message": "Starting syncing SourceAha"}}
{"type": "DEBUG", "message": "parsed YAML into declarative source", "data": {"source_name": "SourceAha", "parsed_config": "{\"version\": \"0.1.0\", \"definitions\": {\"selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"increment_paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}, \"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"base_stream\": {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}}, \"features_stream\": {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"features\", \"path\": \"/features\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, \"products_stream\": {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"products\", \"path\": \"/products\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, \"ideas_stream\": {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"ideas\", \"path\": \"/ideas\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, \"users_stream\": {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"users\", \"path\": \"/users\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, \"goals_stream\": {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"goals\", \"path\": \"/goals\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}}, \"streams\": [{\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"features\", \"path\": \"/features\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"products\", \"path\": \"/products\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"ideas\", \"path\": \"/ideas\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"users\", \"path\": \"/users\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}, {\"retriever\": {\"record_selector\": {\"extractor\": {\"field_pointer\": []}}, \"requester\": {\"url_base\": \"{{ config['url'] }}/api/v1\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"BearerAuthenticator\", \"api_token\": \"{{ config['api_key'] }}\"}}, \"paginator\": {\"type\": \"DefaultPaginator\", \"url_base\": \"{{ config['url'] }}/api/v1\", \"page_size_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"per_page\"}, \"pagination_strategy\": {\"type\": \"PageIncrement\", \"page_size\": 5}, \"page_token_option\": {\"inject_into\": \"request_parameter\", \"field_name\": \"page\"}}}, \"$options\": {\"name\": \"goals\", \"path\": \"/goals\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}], \"check\": {\"stream_names\": [\"features\", \"products\", \"ideas\", \"users\", \"goals\"]}}", "path_to_yaml_file": "aha.yaml"}}
{"type": "DEBUG", "message": "Syncing configured stream: features", "data": {"sync_mode": "SyncMode.full_refresh", "cursor_field": "None", "primary_key": "None"}}
{"type": "DEBUG", "message": "Syncing stream instance: features", "data": {"cursor_field": "[]", "primary_key": ""}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: features "}}
{"type": "DEBUG", "message": "Processing stream slices for features", "data": {"stream_slices": "<generator object SingleSlice.stream_slices at 0x110b004a0>"}}
{"type": "DEBUG", "message": "Processing stream slice", "data": {"slice": "{}"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"request_body": "None", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features?per_page=5", "headers": "{'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Bearer abb74d48e6d2371242eb2a84d1ac2198a64a60b3396d76c35bf8a6dadcc1f0bb'}"}}
{"type": "DEBUG", "message": "Receiving response", "data": {"status": "200", "body": "{\"features\":[{\"id\":\"7161241252554952509\",\"reference_num\":\"DEMO-30\",\"name\":\"Share places of interest in Italy\",\"created_at\":\"2022-11-02T02:33:32.924Z\",\"url\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-30\",\"resource\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-30\",\"product_id\":\"7161241251467958996\"},{\"id\":\"7161241252668494861\",\"reference_num\":\"DEMO-31\",\"name\":\"Replace getting started guide\",\"created_at\":\"2022-11-02T02:33:32.928Z\",\"url\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-31\",\"resource\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-31\",\"product_id\":\"7161241251467958996\"},{\"id\":\"7161241252796906079\",\"reference_num\":\"DEMO-32\",\"name\":\"Syndicate cycling route data\",\"created_at\":\"2022-11-02T02:33:32.933Z\",\"url\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-32\",\"resource\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-32\",\"product_id\":\"7161241251467958996\"},{\"id\":\"7161241252905600008\",\"reference_num\":\"DEMO-33\",\"name\":\"Highlight supporting Local Bike Shops (LBS)\",\"created_at\":\"2022-11-02T02:33:32.938Z\",\"url\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-33\",\"resource\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-33\",\"product_id\":\"7161241251467958996\"},{\"id\":\"7161241253536612514\",\"reference_num\":\"DEMO-18\",\"name\":\"Add search\",\"created_at\":\"2022-11-02T02:33:32.980Z\",\"url\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-18\",\"resource\":\"https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-18\",\"product_id\":\"7161241251467958996\"}],\"pagination\":{\"total_records\":38,\"total_pages\":8,\"current_page\":1}}", "headers": "{'Server': 'openresty', 'Date': 'Thu, 10 Nov 2022 03:46:52 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Permitted-Cross-Domain-Policies': 'none', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE', 'Access-Control-Allow-Headers': 'Content-Type,Authorization,X-Aha-Account,x-datadog-trace-id,x-datadog-parent-id,x-datadog-origin,x-datadog-sampled,x-datadog-sampling-priority', 'Cache-Control': 'no-store', 'Pragma': 'no-cache', 'Expires': 'Fri, 01 Jan 1970 00:00:00 GMT', 'ETag': 'W/\"9f10be957b0256d1a6c9a189ab16f51b\"', 'Set-Cookie': '_aha_t=7164228839588922303; domain=aha.io; path=/; expires=Sun, 10 Nov 2024 03:46:52 GMT; secure; SameSite=Lax', 'X-Request-Id': 'b7119c15-dfda-41e9-9c28-06c79939dd92', 'X-Runtime': '0.140163', 'Strict-Transport-Security': 'max-age=63072000; includeSubdomains;'}"}}
{"type": "RECORD", "record": {"stream": "features", "data": {"features": [{"id": "7161241252554952509", "reference_num": "DEMO-30", "name": "Share places of interest in Italy", "created_at": "2022-11-02T02:33:32.924Z", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-30", "resource": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-30", "product_id": "7161241251467958996"}, {"id": "7161241252668494861", "reference_num": "DEMO-31", "name": "Replace getting started guide", "created_at": "2022-11-02T02:33:32.928Z", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-31", "resource": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-31", "product_id": "7161241251467958996"}, {"id": "7161241252796906079", "reference_num": "DEMO-32", "name": "Syndicate cycling route data", "created_at": "2022-11-02T02:33:32.933Z", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-32", "resource": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-32", "product_id": "7161241251467958996"}, {"id": "7161241252905600008", "reference_num": "DEMO-33", "name": "Highlight supporting Local Bike Shops (LBS)", "created_at": "2022-11-02T02:33:32.938Z", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-33", "resource": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-33", "product_id": "7161241251467958996"}, {"id": "7161241253536612514", "reference_num": "DEMO-18", "name": "Add search", "created_at": "2022-11-02T02:33:32.980Z", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/features/DEMO-18", "resource": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/features/DEMO-18", "product_id": "7161241251467958996"}], "pagination": {"total_records": 38, "total_pages": 8, "current_page": 1}}, "emitted_at": 1668052012594}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 1 records from features stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing features"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceAha runtimes:\nSyncing stream features 0:00:01.516667"}}
{"type": "DEBUG", "message": "Syncing configured stream: products", "data": {"sync_mode": "SyncMode.full_refresh", "cursor_field": "None", "primary_key": "None"}}
{"type": "DEBUG", "message": "Syncing stream instance: products", "data": {"cursor_field": "[]", "primary_key": ""}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: products "}}
{"type": "DEBUG", "message": "Processing stream slices for products", "data": {"stream_slices": "<generator object SingleSlice.stream_slices at 0x110b00120>"}}
{"type": "DEBUG", "message": "Processing stream slice", "data": {"slice": "{}"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"request_body": "None", "url": "https://905c7285-30d2-4419-a9e4-8eef125ce6c1.aha.io/api/v1/products?per_page=5", "headers": "{'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Bearer abb74d48e6d2371242eb2a84d1ac2198a64a60b3396d76c35bf8a6dadcc1f0bb'}"}}
{"type": "DEBUG", "message": "Receiving response", "data": {"status": "200", "body": "{\"products\":[{\"id\":\"7161241251467958996\",\"reference_prefix\":\"DEMO\",\"name\":\"Fredwin Cycling Product (Demo)\",\"product_line\":false,\"created_at\":\"2022-11-02T02:33:32.892Z\",\"workspace_type\":\"product_workspace\"},{\"id\":\"7161241248489048681\",\"reference_prefix\":\"DEMOCO\",\"name\":\"Fredwin Software (Demo)\",\"product_line\":true,\"created_at\":\"2022-11-02T02:33:29.574Z\",\"workspace_type\":\"product_workspace\"},{\"id\":\"7161241233152507326\",\"reference_prefix\":\"COMPANY\",\"name\":\"Test Company\",\"product_line\":true,\"created_at\":\"2022-11-02T02:33:25.670Z\",\"workspace_type\":\"product_workspace\"},{\"id\":\"7161241330809945782\",\"reference_prefix\":\"PROD\",\"name\":\"workspace1\",\"product_line\":false,\"created_at\":\"2022-11-02T02:33:48.409Z\",\"workspace_type\":\"product_workspace\"}],\"pagination\":{\"total_records\":4,\"total_pages\":1,\"current_page\":1}}", "headers": "{'Server': 'openresty', 'Date': 'Thu, 10 Nov 2022 03:46:53 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Permitted-Cross-Domain-Policies': 'none', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE', 'Access-Control-Allow-Headers': 'Content-Type,Authorization,X-Aha-Account,x-datadog-trace-id,x-datadog-parent-id,x-datadog-origin,x-datadog-sampled,x-datadog-sampling-priority', 'Cache-Control': 'no-store', 'Pragma': 'no-cache', 'Expires': 'Fri, 01 Jan 1970 00:00:00 GMT', 'ETag': 'W/\"7a192e8d64caff47a55f075696b4f676\"', 'Set-Cookie': '_aha_t=7164228843885502057; domain=aha.io; path=/; expires=Sun, 10 Nov 2024 03:46:53 GMT; secure; SameSite=Lax', 'X-Request-Id': '47bdf820-4978-422b-8b60-ad66acf2def3', 'X-Runtime': '0.069396', 'Strict-Transport-Security': 'max-age=63072000; includeSubdomains;'}"}}
{"type": "RECORD", "record": {"stream": "products", "data": {"products": [{"id": "7161241251467958996", "reference_prefix": "DEMO", "name": "Fredwin Cycling Product (Demo)", "product_line": false, "created_at": "2022-11-02T02:33:32.892Z", "workspace_type": "product_workspace"}, {"id": "7161241248489048681", "reference_prefix": "DEMOCO", "name": "Fredwin Software (Demo)", "product_line": true, "created_at": "2022-11-02T02:33:29.574Z", "workspace_type": "product_workspace"}, {"id": "7161241233152507326", "reference_prefix": "COMPANY", "name": "Test Company", "product_line": true, "created_at": "2022-11-02T02:33:25.670Z", "workspace_type": "product_workspace"}, {"id": "7161241330809945782", "reference_prefix": "PROD", "name": "workspace1", "product_line": false, "created_at": "2022-11-02T02:33:48.409Z", "workspace_type": "product_workspace"}], "pagination": {"total_records": 4, "total_pages": 1, "current_page": 1}}, "emitted_at": 1668052013715}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 1 records from products stream"}}

[girarda]

@bala-ceg your logs show that only one page of requests was fetched for each stream slice. Additional pages were not requested because the first page contains fewer records than a page can contain.

I also want to point out that the data seems to be in the "products" field of the response, so you should update the extractor's field_pointer to ["products"]

[bala-ceg]

@girarda, thanks for taking a look at this. I have around 30 records, but I still outwards API request is not sending any page number request. I have the exact config in this airbyte low-code tutorial

[girarda]

@bala-ceg the connector in your PR is interpreting the responses as being only one record because it doesn't know where to find the records in the response.

You can use a record selector to extract the records from the response.

Depending on the stream, it will look like:

  selector:
    extractor:
      field_pointer: ["products"]

or

  selector:
    extractor:
      field_pointer: ["features"]

[bala-ceg]

Thanks, @girarda this is an eye-opener for me. I fixed the issue with my connector.