feat: config codegen

[tisonkun]

• This closes feat: Codegen for rust config #4278.
• This refers to idea: Introduce Codegen into OpenDAL #4279.

Let's see if we'd like to go in this way.

[Xuanwo]

I don't find writing in Go appealing, especially since almost all of our dev related scripts are written in python.

[tisonkun]

@Xuanwo think of it a typed Python script. Better than untyped toml strings..

But maybe I can port all these things into Python with Jinja template or sth.

Just no untyped toml, we can write the schema in code.

Newer Python can be typed also.

[tisonkun]

@Xuanwo finished to Python.

[tisonkun]

I don't know what's the idiom to group methods. In Golang, receiver methods can scatter in multiple files. In Python, perhaps we need some design pattern like "XlangConfig(Field)Generator" and take "Config(Field)" a parameter.

[tisonkun]

Follow up -

• Tracking issues to convert other configs in this pattern.
• Work on generate code for Java / Python / ...

cc @Xuanwo @PsiACE

[Xuanwo]

I'm opposed to installing tools like pdm. In fact, I want to avoid adding any new dependencies. We don't need to package/release/dist it. I expect this code generation can be executed with a simple python3 xxx.py, just like all other scripts. Thus I prefer to keep it inside scripts.

Or we can make scripts as a whole new package that managed as a real python project.

[tisonkun]

I want to avoid adding any new dependencies.

This is strange. It means we recreate many utilities by hand.

Code generator isn't such a trivial thing.

[tisonkun]

Or we can make scripts as a whole new package that managed as a real python project.

This is possible. Or call it tools/pytools. I suppose we'd have some handy scripts may be not in Python (or Python is not a good fit).

Firstly I'd make a consensus that write a code generator without any 3rdparty will end up with write every utility by hand and it's even .. stupid.

Then we back to the naming issue about how we call the current "codegen" folder.

[Xuanwo]

Firstly I'd make a consensus that write a code generator without any 3rdparty will end up with write every utility by hand and it's even .. stupid.

Agrees.

Then we back to the naming issue about how we call the current "codegen" folder.

We can merge current scripts and codegen into one tools or odev.

And maybe we can use the same entry point: odev?

Having a odev --help maybe make it easier to discover?

[Xuanwo]

And maybe we can use the same entry point: odev?

This is much furthur on current codebase. We can start with treating tools as a python project first.

[Xuanwo]

I plan to generate more codes for S3Builder. Therefore, I prefer to maintain them separately in each service rather than composing everything into one large config.rs.

[tisonkun]

This is possible. So we keep generate the S3 config to core/src/service/s3/?.

Maybe you have some inputs about the layout. I'm open to this point.

[Xuanwo]

I think we can have core/src/service/${service}/generated.rs. It's also fine to me to have core/src/service/${service}/config.generated.rs

[tisonkun]

scripts/ is written with a mindset used standalone.

Forcely combine it with our current work is strange and extremely dirty.

We're merging two irrelevant things while implementing one of it in a single PR.

If we don't collaborate but rush alone and fix it, it's possible. But now we're trying to review it; then it won't work or just blindly merge.

I don't want to do this.