-
Notifications
You must be signed in to change notification settings - Fork 524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: config codegen #4378
Closed
Closed
feat: config codegen #4378
Changes from all commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
fdf8525
feat: config codegen
tisonkun 42ecf16
first
tisonkun 56f8657
more
tisonkun 8213816
all fields
tisonkun 4d83140
dbg
tisonkun 28dff6d
binary
tisonkun bb3d70a
generate
tisonkun d3d879c
Makefile
tisonkun bef7d29
readme
tisonkun 0444390
force format
tisonkun 907acc7
license header
tisonkun 5861c78
clippy
tisonkun 33febb9
tidy
tisonkun 83b856c
separate files
tisonkun 542a8ab
mod tidy
tisonkun be01a11
clippy
tisonkun 7ca0636
setup python dev env
tisonkun 3d663d6
more setup
tisonkun 8ab9940
data enum
tisonkun 18c72e9
more impl
tisonkun 30f892c
add s3 config
tisonkun dca24e8
tmpl
tisonkun bb769b6
finish tmpl
tisonkun a6218fd
all finished
tisonkun 0c94598
fixup
tisonkun 38c3af9
drop old code
tisonkun c94708b
exp
tisonkun fade1e2
license header
tisonkun ce49212
fix all rules
tisonkun 52bdb2c
docs
tisonkun d4bf7b3
typecheck
tisonkun abc7d30
simplify code
tisonkun c3f266a
Merge branch 'main' into codegen
tisonkun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,162 @@ | ||
# Byte-compiled / optimized / DLL files | ||
__pycache__/ | ||
*.py[cod] | ||
*$py.class | ||
|
||
# C extensions | ||
*.so | ||
|
||
# Distribution / packaging | ||
.Python | ||
build/ | ||
develop-eggs/ | ||
dist/ | ||
downloads/ | ||
eggs/ | ||
.eggs/ | ||
lib/ | ||
lib64/ | ||
parts/ | ||
sdist/ | ||
var/ | ||
wheels/ | ||
share/python-wheels/ | ||
*.egg-info/ | ||
.installed.cfg | ||
*.egg | ||
MANIFEST | ||
|
||
# PyInstaller | ||
# Usually these files are written by a python script from a template | ||
# before PyInstaller builds the exe, so as to inject date/other infos into it. | ||
*.manifest | ||
*.spec | ||
|
||
# Installer logs | ||
pip-log.txt | ||
pip-delete-this-directory.txt | ||
|
||
# Unit test / coverage reports | ||
htmlcov/ | ||
.tox/ | ||
.nox/ | ||
.coverage | ||
.coverage.* | ||
.cache | ||
nosetests.xml | ||
coverage.xml | ||
*.cover | ||
*.py,cover | ||
.hypothesis/ | ||
.pytest_cache/ | ||
cover/ | ||
|
||
# Translations | ||
*.mo | ||
*.pot | ||
|
||
# Django stuff: | ||
*.log | ||
local_settings.py | ||
db.sqlite3 | ||
db.sqlite3-journal | ||
|
||
# Flask stuff: | ||
instance/ | ||
.webassets-cache | ||
|
||
# Scrapy stuff: | ||
.scrapy | ||
|
||
# Sphinx documentation | ||
docs/_build/ | ||
|
||
# PyBuilder | ||
.pybuilder/ | ||
target/ | ||
|
||
# Jupyter Notebook | ||
.ipynb_checkpoints | ||
|
||
# IPython | ||
profile_default/ | ||
ipython_config.py | ||
|
||
# pyenv | ||
# For a library or package, you might want to ignore these files since the code is | ||
# intended to run in multiple environments; otherwise, check them in: | ||
# .python-version | ||
|
||
# pipenv | ||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. | ||
# However, in case of collaboration, if having platform-specific dependencies or dependencies | ||
# having no cross-platform support, pipenv may install dependencies that don't work, or not | ||
# install all needed dependencies. | ||
#Pipfile.lock | ||
|
||
# poetry | ||
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. | ||
# This is especially recommended for binary packages to ensure reproducibility, and is more | ||
# commonly ignored for libraries. | ||
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control | ||
#poetry.lock | ||
|
||
# pdm | ||
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. | ||
#pdm.lock | ||
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it | ||
# in version control. | ||
# https://pdm-project.org/#use-with-ide | ||
.pdm.toml | ||
.pdm-python | ||
.pdm-build/ | ||
|
||
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm | ||
__pypackages__/ | ||
|
||
# Celery stuff | ||
celerybeat-schedule | ||
celerybeat.pid | ||
|
||
# SageMath parsed files | ||
*.sage.py | ||
|
||
# Environments | ||
.env | ||
.venv | ||
env/ | ||
venv/ | ||
ENV/ | ||
env.bak/ | ||
venv.bak/ | ||
|
||
# Spyder project settings | ||
.spyderproject | ||
.spyproject | ||
|
||
# Rope project settings | ||
.ropeproject | ||
|
||
# mkdocs documentation | ||
/site | ||
|
||
# mypy | ||
.mypy_cache/ | ||
.dmypy.json | ||
dmypy.json | ||
|
||
# Pyre type checker | ||
.pyre/ | ||
|
||
# pytype static type analyzer | ||
.pytype/ | ||
|
||
# Cython debug symbols | ||
cython_debug/ | ||
|
||
# PyCharm | ||
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can | ||
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore | ||
# and can be added to the global gitignore or merged into this file. For a more nuclear | ||
# option (not recommended) you can uncomment the following to ignore the entire idea folder. | ||
#.idea/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# Codegen | ||
|
||
## Generate | ||
|
||
### Generate service config files | ||
|
||
Run the following command to generate service config files: | ||
|
||
```bash | ||
pdm genconfig | ||
``` | ||
|
||
## Develop | ||
|
||
### Check code style and typing | ||
|
||
Run the following command to invoke linter, formatter, and type checker. | ||
|
||
```bash | ||
pdm check | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
import subprocess | ||
from pathlib import Path | ||
|
||
from jinja2 import Environment, FileSystemLoader | ||
|
||
from codegen.config import S3 | ||
|
||
if __name__ == '__main__': | ||
codegen = Path(__file__).resolve().parent.parent | ||
env = Environment( | ||
loader=FileSystemLoader(codegen / 'template'), | ||
) | ||
tmpl = env.get_template("config.rs.j2") | ||
|
||
opendal = codegen.parent | ||
rust = opendal / 'core' / 'src' / 'services' / 'config.rs' | ||
with rust.open('w') as f: | ||
assert tmpl.filename is not None | ||
template = Path(tmpl.filename).relative_to(opendal) | ||
output = rust.relative_to(opendal) | ||
print(f'Generating Rust files from templates in {template} to {output}') | ||
f.write(tmpl.render(configs=[S3])) | ||
|
||
subprocess.run(['rustfmt', rust]) |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm opposed to installing tools like
pdm
. In fact, I want to avoid adding any new dependencies. We don't need to package/release/dist it. I expect this code generation can be executed with a simplepython3 xxx.py
, just like all other scripts. Thus I prefer to keep it insidescripts
.Or we can make
scripts
as a whole new package that managed as a real python project.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is strange. It means we recreate many utilities by hand.
Code generator isn't such a trivial thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is possible. Or call it
tools
/pytools
. I suppose we'd have some handy scripts may be not in Python (or Python is not a good fit).Firstly I'd make a consensus that write a code generator without any 3rdparty will end up with write every utility by hand and it's even .. stupid.
Then we back to the naming issue about how we call the current "codegen" folder.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agrees.
We can merge current
scripts
andcodegen
into onetools
orodev
.And maybe we can use the same entry point:
odev
?Having a
odev --help
maybe make it easier to discover?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is much furthur on current codebase. We can start with treating
tools
as a python project first.