Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(core): adopt WebHDFS LISTSTATUS_BATCH for better performance #2499

Merged
merged 12 commits into from
Jun 21, 2023
Merged

feat(core): adopt WebHDFS LISTSTATUS_BATCH for better performance #2499

merged 12 commits into from
Jun 21, 2023

Conversation

morristai
Copy link
Member

Description

Background

Use the WebHDFS Iteratively_List_a_Directory API in WebHDFS as the default listing behavior, and fallback to One-time Listing API if the server doesn't support batch listing.

@morristai morristai marked this pull request as draft June 20, 2023 00:21
Xuanwo
Xuanwo reviewed Jun 20, 2023
View reviewed changes
core/src/services/webhdfs/backend.rs Outdated Show resolved Hide resolved
core/src/services/webhdfs/backend.rs Outdated Show resolved Hide resolved
@morristai morristai force-pushed the feat/adopt_webhdfs_liststatus_batch branch from a9ac40e to 3f044d6 Compare June 20, 2023 20:20
Xuanwo
Xuanwo reviewed Jun 21, 2023
View reviewed changes
core/src/services/webhdfs/backend.rs Outdated Show resolved Hide resolved
core/src/services/webhdfs/backend.rs Outdated Show resolved Hide resolved
core/src/services/webhdfs/message.rs Outdated Show resolved Hide resolved
@morristai morristai marked this pull request as ready for review June 21, 2023 04:08
Xuanwo
Xuanwo approved these changes Jun 21, 2023
View reviewed changes
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@Xuanwo Xuanwo merged commit 04852a4 into apache:main Jun 21, 2023
@morristai
Copy link
Member Author

BTW, same dilemma on whether to use #[derive(default)] or impl Default. My first implementation used the derive macro, but it required four other sub struct/enum to implement the default macro. Anyway, I've learned a lot from these PR practices. I'll ask for help when I'm stuck again.

@Xuanwo
Copy link
Member

Xuanwo commented Jun 21, 2023

I'll ask for help when I'm stuck again.

Feel free to ask for help. I am willing to answer any questions you may have!

@PsiACE PsiACE mentioned this pull request Jun 27, 2023
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Xuanwo Xuanwo Xuanwo approved these changes

@PsiACE PsiACE Awaiting requested review from PsiACE

@ClSlaid ClSlaid Awaiting requested review from ClSlaid

@sundy-li sundy-li Awaiting requested review from sundy-li

Assignees
No one assigned
Labels
releases-note/feat
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
2 participants
@morristai @Xuanwo