Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ballista: Implement configuration mechanism #19

Open
andygrove opened this issue Jul 5, 2021 · 0 comments
Open

Ballista: Implement configuration mechanism #19

andygrove opened this issue Jul 5, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@andygrove
Copy link
Member

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There is currently no way to specify configuration parameters such as default partition count in Ballista.

Describe the solution you'd like
As a user, I would like to be able to set configurations such as default partition count and have these settings propagated to the scheduler and executors via protobuf.

Describe alternatives you've considered
None

Additional context
None
@andygrove andygrove added enhancement New feature or request ballista labels Jul 5, 2021
@andygrove andygrove self-assigned this Jul 5, 2021
@andygrove andygrove mentioned this issue Jul 6, 2021
Closed
This was referenced Feb 18, 2022
Closed
Closed
@andygrove andygrove removed their assignment May 2, 2022
@andygrove andygrove transferred this issue from apache/datafusion May 19, 2022
thinkharderdev added a commit that referenced this issue Jan 21, 2023
@thinkharderdev @mpurins-coralogix
Cluster state refactor part 1 (#560)
93cf14f 
* Customize session builder

* Add setter for executor slots policy

* Construct Executor with functions

* Add queued and completed timestamps to successful job status

* Add public methods to SchedulerServer

* Public method for getting execution graph

* Public method for stage metrics

* Use node-level local limit (#20)

* Use node-level local limit

* serialize limit in shuffle writer

* Revert "Merge pull request #19 from coralogix/sc-5792"

This reverts commit 08140ef, reversing
changes made to a7f1384.

* add log

* make sure we don't forget limit for shuffle writer

* update accum correctly and try to break early

* Check local limit accumulator before polling for more data

* fix build

Co-authored-by: Martins Purins <[email protected]>

* configure_me_codegen retroactively reserved on our `bind_host` parame… (#520)

* configure_me_codegen retroactively reserved on our `bind_host` parameter name

* Add label and pray

* Add more labels why not

* Add ClusterState trait

* Refactor slightly for clarity

* Revert "Use node-level local limit (#20)"

This reverts commit ff96bcd.

* Revert "Public method for stage metrics"

This reverts commit a802315.

* Revert "Public method for getting execution graph"

This reverts commit 490bda5.

* Revert "Add public methods to SchedulerServer"

This reverts commit 5ad27c0.

* Revert "Add queued and completed timestamps to successful job status"

This reverts commit c615fce.

* Revert "Construct Executor with functions"

This reverts commit 24d4830.

* Always forget the apache header

Co-authored-by: Martins Purins <[email protected]>
Co-authored-by: Brent Gardner <[email protected]>
thinkharderdev added a commit that referenced this issue Feb 7, 2023
@thinkharderdev @mpurins-coralogix
Use node-level local limit (#20)
ff96bcd 
* Use node-level local limit

* serialize limit in shuffle writer

* Revert "Merge pull request #19 from coralogix/sc-5792"

This reverts commit 08140ef, reversing
changes made to a7f1384.

* add log

* make sure we don't forget limit for shuffle writer

* update accum correctly and try to break early

* Check local limit accumulator before polling for more data

* fix build

Co-authored-by: Martins Purins <[email protected]>
andygrove pushed a commit that referenced this issue Feb 11, 2023
@thinkharderdev @mpurins-coralogix
Cluster state refactor Part 2 (#658)
e7f8774 
* Customize session builder

* Add setter for executor slots policy

* Construct Executor with functions

* Add queued and completed timestamps to successful job status

* Add public methods to SchedulerServer

* Public method for getting execution graph

* Public method for stage metrics

* Use node-level local limit (#20)

* Use node-level local limit

* serialize limit in shuffle writer

* Revert "Merge pull request #19 from coralogix/sc-5792"

This reverts commit 08140ef, reversing
changes made to a7f1384.

* add log

* make sure we don't forget limit for shuffle writer

* update accum correctly and try to break early

* Check local limit accumulator before polling for more data

* fix build

Co-authored-by: Martins Purins <[email protected]>

* configure_me_codegen retroactively reserved on our `bind_host` parame… (#520)

* configure_me_codegen retroactively reserved on our `bind_host` parameter name

* Add label and pray

* Add more labels why not

* Add ClusterState trait

* Refactor slightly for clarity

* Revert "Use node-level local limit (#20)"

This reverts commit ff96bcd.

* Revert "Public method for stage metrics"

This reverts commit a802315.

* Revert "Public method for getting execution graph"

This reverts commit 490bda5.

* Revert "Add public methods to SchedulerServer"

This reverts commit 5ad27c0.

* Revert "Add queued and completed timestamps to successful job status"

This reverts commit c615fce.

* Revert "Construct Executor with functions"

This reverts commit 24d4830.

* Always forget the apache header

* WIP

* Implement JobState

* Tests and fixes

* do not hold ref across await point

* Fix clippy warnings

* Fix tomlfmt github action

* uncomment test

---------

Co-authored-by: Martins Purins <[email protected]>
Co-authored-by: Brent Gardner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@andygrove