Dynamically Linked Library in Presto CPP #24330
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
prestodb-ci
requested review from
a team,
pdabre12 and
psnv03
and removed request for
a team
czentgr
reviewed
Feb 3, 2025
| @@ -76,7 +76,8 @@ target_link_libraries( | |||
| ${FOLLY_WITH_DEPENDENCIES} | |||
| ${GLOG} | |||
| ${GFLAGS_LIBRARIES} | |||
| pthread) | |||
| pthread | |||
| velox_dynamic_function_loader) | |||
There was a problem hiding this comment.
Move this before the velox_encode library.
| @@ -89,6 +90,7 @@ set_property(TARGET presto_server_lib PROPERTY JOB_POOL_LINK | |||
| presto_link_job_pool) | |||
|
|
|||
| add_executable(presto_server PrestoMain.cpp) | |||
| target_link_options(presto_server BEFORE PUBLIC "-Wl,-export-dynamic") | |||
There was a problem hiding this comment.
Can we add a comment here why we need these flags?
| const fs::path path(systemConfig->pluginDir()); | ||
| PRESTO_STARTUP_LOG(INFO) << path; | ||
| std::error_code | ||
| ec; // For using the non-throwing overloads of functions below. |
There was a problem hiding this comment.
We don;t need this comment here and so can fix up the odd formatting.
| void PrestoServer::registerDynamicFunctions() { | ||
| auto systemConfig = SystemConfig::instance(); | ||
| if (!systemConfig->pluginDir().empty()) { | ||
| // if it is a valid directory, traverse and call dynamic function loader |
There was a problem hiding this comment.
Please make sure the comments are full sentences beginning with capitalization etc.
| auto dirEntryPath = dirEntry.path(); | ||
| if (!fs::is_directory(dirEntry, ec) && | ||
| extensions.find(dirEntryPath.extension()) != extensions.end()) { | ||
| facebook::velox::loadDynamicLibrary(dirEntryPath.c_str()); |
There was a problem hiding this comment.
facebook is not needed here because we are already in the facebook namespace.
steveburnett
requested changes
Feb 4, 2025
| @@ -0,0 +1,17 @@ | |||
| # Dynamic Loading of Presto Cpp Extensions | |||
There was a problem hiding this comment.
Suggested change
| # Dynamic Loading of Presto Cpp Extensions | |
| # Dynamic Loading of Presto CPP Extensions |
| @@ -0,0 +1,17 @@ | |||
| # Dynamic Loading of Presto Cpp Extensions | |||
| This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library. | |||
There was a problem hiding this comment.
Suggested change
| This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library. | |
| This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library. |
| This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library. | ||
| ## Getting started | ||
| 1. Create a cpp file for your dynamic library | ||
| For dynamically loaded function registration, the format followed is mirrored of that of built-in function registration with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here. |
There was a problem hiding this comment.
Suggested change
| For dynamically loaded function registration, the format followed is mirrored of that of built-in function registration with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here. | |
| For dynamically loaded function registration, the format is similar to that of built-in function registration, with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here. |
| # Dynamic Loading of Presto Cpp Extensions | ||
| This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library. | ||
| ## Getting started | ||
| 1. Create a cpp file for your dynamic library |
There was a problem hiding this comment.
Suggested change
| 1. Create a cpp file for your dynamic library | |
| 1. Create a cpp file for your dynamic library | |
Add a blank line after the first sentence in each step to avoid this line running into the next line. See screenshot of what the current formatting looks like.
| ## Getting started | ||
| 1. Create a cpp file for your dynamic library | ||
| For dynamically loaded function registration, the format followed is mirrored of that of built-in function registration with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here. | ||
| 2. Register functions dynamically by creating .dylib or .so shared libraries and dropping them in a plugin directory |
There was a problem hiding this comment.
Suggested change
| 2. Register functions dynamically by creating .dylib or .so shared libraries and dropping them in a plugin directory | |
| 2. Register functions dynamically by creating .dylib or .so shared libraries and dropping them in a plugin directory | |
| add_library(name_of_dynamic_fn SHARED TestFunction.cpp) | ||
| target_link_libraries(name_of_dynamic_fn PRIVATE fmt::fmt Folly::folly gflags::gflags) | ||
| ``` | ||
| 3. In the Prestissimo worker's config.properties file, set the plugin.dir property |
There was a problem hiding this comment.
Suggested change
| 3. In the Prestissimo worker's config.properties file, set the plugin.dir property | |
| 3. In the Prestissimo worker's config.properties file, set the plugin.dir property | |
| ``` | ||
| plugin.dir="User\Test\Path\plugin" | ||
| ``` | ||
| 4. When the worker or the sidecar process starts, it will scan the plugin directory and attempt to dynamically load all shared libraries |
There was a problem hiding this comment.
Suggested change
| 4. When the worker or the sidecar process starts, it will scan the plugin directory and attempt to dynamically load all shared libraries | |
| When the worker or the sidecar process starts, it scans the plugin directory and attempts to dynamically load all shared libraries. |
I don't think of this as part of the steps to configure, it's what happens when things are run using the config in steps 1-3.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Depends on facebookincubator/velox#11439 in the Velox space
and based off of the following PR: https://github.com/facebookincubator/velox/pull/1005/files
Motivation and Context
Having these changes will enable users to register custom functions dynamically without requiring a fork of Prestissimo.
Impact
This extends Prestissimo functionality to include dynamic loading of functions, types, connectors, etc.
Test Plan
Unit tested. and Manually end to end tested the changes.
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.
If release note is NOT required, use: