Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: taskmanager add kubernetes backend #3147

Merged

Conversation

tobegit3hub
Copy link
Collaborator

Usage:

Prepare K8S cluster and config. Set taskmanager backend of K8S.

spark.master=k8s

If we use local K8S, we can access local OpenMLDB and local filesystem. Here is the example config.

zookeeper.cluster=10.120.2.40:2181
zookeeper.root_path=/openmldb

offline.data.prefix=file:///Users/tobe/code/PortableOpenMLDB/udisk/openmldb_offline_data/

spark.master=k8s

Known issues:

  • Do not support Hadoop/HDFS
  • Job status is not updated, blocking mode is not supported yet

@tobegit3hub tobegit3hub self-assigned this Mar 15, 2023
@github-actions github-actions bot added batch-engine openmldb batch(offline) engine task-manager openmldb taskmanager labels Mar 15, 2023
@tobegit3hub tobegit3hub changed the title feat: ttaskmanager add kubernetes backend feat: taskmanager add kubernetes backend Mar 15, 2023
@codecov
Copy link

codecov bot commented Mar 15, 2023

Codecov Report

Patch coverage has no change and project coverage change: -3.38 ⚠️

Comparison is base (9f7e245) 79.04% compared to head (76318a2) 75.66%.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3147      +/-   ##
============================================
- Coverage     79.04%   75.66%   -3.38%     
- Complexity        0      393     +393     
============================================
  Files           512      673     +161     
  Lines        115515   124798    +9283     
  Branches          0     1139    +1139     
============================================
+ Hits          91305    94429    +3124     
- Misses        24210    30149    +5939     
- Partials          0      220     +220     
Impacted Files Coverage Δ
...adigm/openmldb/batchjob/util/OpenmldbJobUtil.scala 0.00% <0.00%> (ø)
...openmldb/taskmanager/config/TaskManagerConfig.java 0.00% <0.00%> (ø)
...nmldb/taskmanager/server/impl/TaskManagerImpl.java 0.00% <ø> (ø)
...openmldb/taskmanager/OpenmldbBatchjobManager.scala 0.00% <0.00%> (ø)
...radigm/openmldb/taskmanager/k8s/K8sJobConfig.scala 0.00% <0.00%> (ø)
...adigm/openmldb/taskmanager/k8s/K8sJobManager.scala 0.00% <0.00%> (ø)
...aradigm/openmldb/taskmanager/k8s/K8sYamlUtil.scala 0.00% <0.00%> (ø)

... and 167 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@github-actions
Copy link
Contributor

SDK Test Report

  95 files    95 suites   2m 7s ⏱️
185 tests 175 ✔️ 10 💤 0
231 runs  221 ✔️ 10 💤 0

Results for commit 76318a2.

@github-actions
Copy link
Contributor

Linux Test Report

       56 files       230 suites   1h 14m 52s ⏱️
11 923 tests 11 916 ✔️ 7 💤 0
16 924 runs  16 917 ✔️ 7 💤 0

Results for commit 76318a2.

@dl239
Copy link
Collaborator

dl239 commented Mar 16, 2023

will add docs in next pr?

@tobegit3hub tobegit3hub merged commit d085ac8 into 4paradigm:main Mar 16, 2023
@lumianph lumianph added alpha this feature is not fully tested or functional and removed alpha this feature is not fully tested or functional labels Apr 25, 2023
@lqy222 lqy222 mentioned this pull request May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alpha this feature is not fully tested or functional batch-engine openmldb batch(offline) engine task-manager openmldb taskmanager
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support kubernetes as taskmanager backend
4 participants