Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add physicalWrittenBytes to PlanNodeStats #8213

Closed
wants to merge 4 commits into from
Closed

Add physicalWrittenBytes to PlanNodeStats #8213

wants to merge 4 commits into from

Conversation

JkSelf
Copy link
Collaborator

@JkSelf JkSelf commented Jan 2, 2024 •
edited by mbasmanova
Loading

The physicalWrittenBytes metric is available in the OperatorStats class, but not
in the PlanNodeStats. When accessing metrics from Task::taskStats(), we are
unable to retrieve the physicalWrittenBytes' value.

Ad physicalWrittenBytes metric to PlanNodeStats.

@netlify Netlify
Copy link

netlify bot commented Jan 2, 2024 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 8c0bbb5
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/659746d7b999ec0008a7f11a

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 2, 2024
@JkSelf JkSelf mentioned this pull request Jan 3, 2024
Merged
@JkSelf
Copy link
Collaborator Author

JkSelf commented Jan 3, 2024

@mbasmanova @rui-mo Can you help to review? Thanks.

@mbasmanova mbasmanova changed the title Add physicalWrittenBytes into the PlanNodeStats Add physicalWrittenBytes to PlanNodeStats Jan 3, 2024
mbasmanova
mbasmanova reviewed Jan 3, 2024
View reviewed changes
velox/exec/tests/TableWriteTest.cpp Outdated

auto planStats = exec::toPlanStats(task->taskStats());
auto& stats = planStats.at(tableWriteNodeId_);
for (const auto& entry : stats.operatorStats) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop and if check inside are not needed. You should be able to access stats.physicalWrittenBytes directly.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mbasmanova Updated.

marin-ma pushed a commit to oap-project/velox that referenced this pull request Jan 4, 2024
@JkSelf @marin-ma
Add physicalWrittenBytes into the PlanNodeStats (facebookincubator#8213)
0b234d2 
Pass spark 3.4 unit test when enabling native parquet write (#466)
xiaoxmeng
xiaoxmeng approved these changes Jan 4, 2024
View reviewed changes
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JkSelf LGTM after addressing comments from @mbasmanova. Thanks!

mbasmanova
mbasmanova approved these changes Jan 5, 2024
View reviewed changes
Copy link
Contributor

@mbasmanova mbasmanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@facebook-github-bot
Copy link
Contributor

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@mbasmanova
Copy link
Contributor

@kgpai @majetideepak @assignUser There is a CI failure. Would you know how to resolve it?

Resolving dl.min.io (dl.min.io)... 178.128.69.202, 138.68.11.125
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
ERROR: The certificate of 'dl.min.io' is not trusted.
ERROR: The certificate of 'dl.min.io' has expired.
The certificate has expired

@kgpai
Copy link
Contributor

kgpai commented Jan 5, 2024

@majetideepak we should build minio as part of the docker script , should be as simple as moving this :

 wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20220526054841.0.0.x86_64.rpm
rpm -i minio-20220526054841.0.0.x86_64.rpm
rm minio-20220526054841.0.0.x86_64.rpm

in setup-adapters.sh

I will create a PR for that..

@kgpai
Copy link
Contributor

kgpai commented Jan 5, 2024

Actually I realize that we use the same docker image for the adapter jobs, moving it to setup-adapters might not really help and we need to create a new docker image for this case.

@kgpai
Copy link
Contributor

kgpai commented Jan 5, 2024

Heres PR #8270 addressing the docker image creation : #8270 . Note that we still need dl.min.io to have a valid cert so we can build it, but once they do it we shouldnt have to wget the rpm on every build.

marin-ma pushed a commit to oap-project/velox that referenced this pull request Jan 5, 2024
@JkSelf @marin-ma
Add physicalWrittenBytes into the PlanNodeStats (facebookincubator#8213)
7f736d4 
Pass spark 3.4 unit test when enabling native parquet write (#466)
marin-ma pushed a commit to oap-project/velox that referenced this pull request Jan 5, 2024
@JkSelf @marin-ma
Add physicalWrittenBytes into the PlanNodeStats (facebookincubator#8213)
e949717 
Pass spark 3.4 unit test when enabling native parquet write (#466)
@facebook-github-bot
Copy link
Contributor

@mbasmanova merged this pull request in e5355f3.

@conbench-facebook conbench-facebook
Copy link

Conbench analyzed the 1 benchmark run on commit e5355f36.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mbasmanova mbasmanova mbasmanova approved these changes

@xiaoxmeng xiaoxmeng xiaoxmeng approved these changes

@kewang1024 kewang1024 Awaiting requested review from kewang1024

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@JkSelf @facebook-github-bot @mbasmanova @kgpai @xiaoxmeng