Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extending arrow to include python build modules #1977

Merged
merged 2 commits into from
Feb 2, 2023

Conversation

nkarpov
Copy link
Contributor

@nkarpov nkarpov commented Jan 31, 2023

Feature or Bugfix

Feature

Detail

Expand PyArrow build to include dependencies found in [features-python](https://github.com/apache/arrow/blob/master/cpp/CMakePresets.json#L134-L147) presets in Arrow build.

This change increases the size of the layer by 0.76mb

current published layer:
-rw-r--r--@   1 nick.karpov  staff    48176905 Jan 26 17:32 awswrangler-layer-2.19.0-py3.7.zip

new layer:
-rw-r--r--     1 nick.karpov  staff  48938154 Jan 31 08:55 awswrangler-layer-2.19.0-py3.7.zip

The goal of this PR is to have this base Lambda layer provide all the necessary arrow dependencies for deltalake since the layer has chosen to include PyArrow already(it could also be provided in its own separate layer in theory). I've used the python-features preset as a basis for the changes to bring the pyarrow build in this layer closer to the published pip install pyarrow package - likely other future packages can also take advantage of this.

Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: f96c220
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@dennyglee
Copy link

cc @fvaleye FYI

@kukushking
Copy link
Contributor

0.76mb doesn't look bad at all 👍

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 9cef1e2
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido jaidisido linked an issue Feb 2, 2023 that may be closed by this pull request
@jaidisido jaidisido added this to the 2.20.0 milestone Feb 2, 2023
@jaidisido jaidisido added enhancement New feature or request minor release Will be addressed in the next minor release labels Feb 2, 2023
@jaidisido jaidisido merged commit c9b5871 into aws:main Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request minor release Will be addressed in the next minor release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Expand AWS Lambda layer pyarrow build?
5 participants