Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: config codegen #4378

Closed
wants to merge 33 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions codegen/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm-project.org/#use-with-ide
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
21 changes: 21 additions & 0 deletions codegen/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Codegen

## Generate

### Generate service config files

Run the following command to generate service config files:

```bash
pdm genconfig
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm opposed to installing tools like pdm. In fact, I want to avoid adding any new dependencies. We don't need to package/release/dist it. I expect this code generation can be executed with a simple python3 xxx.py, just like all other scripts. Thus I prefer to keep it inside scripts.

Or we can make scripts as a whole new package that managed as a real python project.

Copy link
Member Author

@tisonkun tisonkun Mar 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to avoid adding any new dependencies.

This is strange. It means we recreate many utilities by hand.

Code generator isn't such a trivial thing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we can make scripts as a whole new package that managed as a real python project.

This is possible. Or call it tools/pytools. I suppose we'd have some handy scripts may be not in Python (or Python is not a good fit).

Firstly I'd make a consensus that write a code generator without any 3rdparty will end up with write every utility by hand and it's even .. stupid.

Then we back to the naming issue about how we call the current "codegen" folder.

Copy link
Member

@Xuanwo Xuanwo Mar 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Firstly I'd make a consensus that write a code generator without any 3rdparty will end up with write every utility by hand and it's even .. stupid.

Agrees.

Then we back to the naming issue about how we call the current "codegen" folder.

We can merge current scripts and codegen into one tools or odev.

And maybe we can use the same entry point: odev?

Having a odev --help maybe make it easier to discover?

Copy link
Member

@Xuanwo Xuanwo Mar 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And maybe we can use the same entry point: odev?

This is much furthur on current codebase. We can start with treating tools as a python project first.

```

## Develop

### Check code style and typing

Run the following command to invoke linter, formatter, and type checker.

```bash
pdm check
```
41 changes: 41 additions & 0 deletions codegen/bin/genconfig.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

import subprocess
from pathlib import Path

from jinja2 import Environment, FileSystemLoader

from codegen.config import S3

if __name__ == '__main__':
codegen = Path(__file__).resolve().parent.parent
env = Environment(
loader=FileSystemLoader(codegen / 'template'),
)
tmpl = env.get_template("config.rs.j2")

opendal = codegen.parent
rust = opendal / 'core' / 'src' / 'services' / 'config.rs'
with rust.open('w') as f:
assert tmpl.filename is not None
template = Path(tmpl.filename).relative_to(opendal)
output = rust.relative_to(opendal)
print(f'Generating Rust files from templates in {template} to {output}')
f.write(tmpl.render(configs=[S3]))

subprocess.run(['rustfmt', rust])
117 changes: 117 additions & 0 deletions codegen/pdm.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading