Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refining example, add utilities, and fix xdist test error #794

Merged
merged 31 commits into from
Nov 23, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
f70cd95
Fix xdist test error. Also make a small cleanup some codes
loomlike Oct 28, 2022
8772a2b
Merge branch 'main' into jumin/fix_xdist
loomlike Oct 28, 2022
990208b
Revert "Revert 756 (#798)"
loomlike Oct 31, 2022
9c1b93e
Merge branch 'jumin/fix_xdist' into jumin/revert_revert756
loomlike Oct 31, 2022
1fab5f2
revert 798 (revert756 - example notebook refactor). Also add job_util…
loomlike Nov 1, 2022
6a4e8a1
Merge pull request #1 from loomlike/jumin/revert_revert756
loomlike Nov 1, 2022
15f4939
Update test_azure_spark_e2e.py
blrchen Nov 1, 2022
26b7a0d
Fix doc dead links (#805)
blrchen Nov 1, 2022
8e401b4
Improve UI experience and clean up ui code warnings (#801)
Fendoe Nov 1, 2022
afd9309
Add release instructions for Release Candidate (#809)
blrchen Nov 1, 2022
8899f18
Bump version to 0.9.0-rc1 (#810)
blrchen Nov 1, 2022
572d762
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 1, 2022
c4cd2ec
Merge branch 'jumin/revert_revert756' into jumin/fix_xdist
loomlike Nov 1, 2022
995f509
Fix tests to use mocks and fix get_result_df's databricks behavior
loomlike Nov 2, 2022
6198506
fix tem file to dir
loomlike Nov 3, 2022
38aa149
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 3, 2022
ae9095c
checkout the feature_derivations.py from main (it was temporally chan…
loomlike Nov 3, 2022
59bd65c
Remove old databricks sample notebook. Change pip install feathr from…
loomlike Nov 3, 2022
125cc3a
Fix config and get_result_df for synapse
loomlike Nov 7, 2022
6fc93eb
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 7, 2022
1f3894a
Fix generate_config to accept all the feathr env var config name
loomlike Nov 8, 2022
8a610ac
Add more pytests
loomlike Nov 9, 2022
f6f1587
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 9, 2022
4c50485
Use None as default dataformat in the job_utils. Instead, set 'avro' …
loomlike Nov 9, 2022
c049958
Change feathr client to mocked object
loomlike Nov 10, 2022
56974db
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 15, 2022
7d3f8be
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 18, 2022
7fcffba
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 21, 2022
190377c
Change timeout to 1000s in the notebook
loomlike Nov 22, 2022
1d93b40
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 23, 2022
ca5d642
Merge branch 'main' into jumin/fix_xdist
loomlike Nov 23, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/pull_request_push_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ on:
- "docs/**"
- "ui/**"
- "**/README.md"

schedule:
# Runs daily at 1 PM UTC (9 PM CST), will send notification to TEAMS_WEBHOOK
- cron: '00 13 * * *'
Expand Down Expand Up @@ -127,7 +127,7 @@ jobs:
SQL1_USER: ${{secrets.SQL1_USER}}
SQL1_PASSWORD: ${{secrets.SQL1_PASSWORD}}
run: |
# run only test with databricks. run in 4 parallel jobs
# run only test with databricks. run in 6 parallel jobs
pytest -n 6 --cov-report term-missing --cov=feathr_project/feathr feathr_project/test --cov-config=.github/workflows/.coveragerc_db
azure_synapse_test:
# might be a bit duplication to setup both the azure_synapse test and databricks test, but for now we will keep those to accelerate the test speed
Expand Down Expand Up @@ -195,7 +195,7 @@ jobs:
SQL1_PASSWORD: ${{secrets.SQL1_PASSWORD}}
run: |
# skip databricks related test as we just ran the test; also seperate databricks and synapse test to make sure there's no write conflict
# run in 4 parallel jobs to make the time shorter
# run in 6 parallel jobs to make the time shorter
pytest -n 6 --cov-report term-missing --cov=feathr_project/feathr feathr_project/test --cov-config=.github/workflows/.coveragerc_sy

local_spark_test:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -213,3 +213,6 @@ null/*
project/.bloop
metals.sbt
.bsp/sbt.json

# Feathr output debug folder
**/debug/
6 changes: 3 additions & 3 deletions docs/dev_guide/new_contributor_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,19 @@ parent: Feathr Developer Guides

# What can I contribute?
All forms of contributions are welcome, including and not limited to:
* Improve or contribute new [notebook samples](https://github.com/feathr-ai/feathr/tree/main/feathr_project/feathrcli/data/feathr_user_workspace)
* Improve or contribute new [notebook samples](https://github.com/feathr-ai/feathr/tree/main/docs/samples)
* Add tutorial, blog posts, tech talks etc
* Increase media coverage and exposure
* Improve user-facing documentation or developer-facing documentation
* Add testing code
* Add testing code
* Add new features
* Refactor and improve architecture
* For any other forms of contribution and collaboration, don't hesitate to reach out to us.

# I am interested, how can I start?
If you are new to this project, we recommend start with [`good-first-issue`](https://github.com/feathr-ai/feathr/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).

The issues are also labled with what types of programming language the task need.
The issues are also labled with what types of programming language the task need.
* [`good-first-issue` and `Python`](https://github.com/feathr-ai/feathr/issues?q=is%3Aopen+label%3A%22good+first+issue%22+label%3Apython)
* [`good-first-issue` and `Scala`](https://github.com/feathr-ai/feathr/issues?q=is%3Aopen+label%3A%22good+first+issue%22+label%3Ascala)
* [`good-first-issue` and `Java`](https://github.com/feathr-ai/feathr/issues?q=is%3Aopen+label%3A%22good+first+issue%22+label%3Ajava)
Expand Down
2 changes: 1 addition & 1 deletion docs/quickstart_synapse.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Feathr has native cloud integration. Here are the steps to use Feathr on Azure:

1. Follow the [Feathr ARM deployment guide](https://feathr-ai.github.io/feathr/how-to-guides/azure-deployment-arm.html) to run Feathr on Azure. This allows you to quickly get started with automated deployment using Azure Resource Manager template. Alternatively, if you want to set up everything manually, you can checkout the [Feathr CLI deployment guide](https://feathr-ai.github.io/feathr/how-to-guides/azure-deployment-cli.html) to run Feathr on Azure. This allows you to understand what is going on and set up one resource at a time.

2. Once the deployment is complete,run the Feathr Jupyter Notebook by clicking this button: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/feathr-ai/feathr/main?labpath=feathr_project%2Ffeathrcli%2Fdata%2Ffeathr_user_workspace%2Fnyc_driver_demo.ipynb).
2. Once the deployment is complete,run the Feathr Jupyter Notebook by clicking this button: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/feathr-ai/feathr/main?labpath=docs%2Fsamples%2Fnyc_taxi_demo.ipynb).
3. You only need to change the specified `Resource Prefix`.

## Step 2: Install Feathr
Expand Down
Loading