DDL project scope and goals

[flaxed]

This project started with the realization that good tooling to work with data definitions does not exist in a open and shared form. Instead teams have to reinvent their own pipelines from the ground up, by defining the DDL spec, the parser and all the tool ecosystem around the format.

Since the inception of the idea of a shared format, we’ve had good discussions, and good progress on defining what the file format will be.

But there’s also some features that will have a direct impact on what the scope of the project can be now and can evolve into.

One thing I personally believe is the value of a format comes not just from its syntax, but more importantly from it’s ecosystem of tools.

Trying to do a summary of all ideas put forward during the various discussions for the format, I propose we aim to build an ecosystem with the following content:

Spec

• A well defined spec that allows any one to support the format with their own parser if they choose or require to.
• Example collection and test suit to help on the development of parsers and tools consuming the format.

Libraries

• Provide parser libraries for all major languages used in gamedev: C, C++, Rust, Python, C#
• Each library should provide:
  • Output of the file AST
  • Output of the information contained in the file, with types, attributes, fields, scopes, and all other information, ready to be consumed in an easy manner by user applications.

Adoptability

There is no point in going through all the effort of creating this ecosystem, if it can't be adopted.
Either by lacking features, or by the features being implemented in a way that is incompatible with the existing pipeline.

Being feature rich will be a community effort, with both users providing feedback on their usecases, and the format mantainers making an effort to support all relevant cases.

Creating tools that don't clash directly with each user existing pipeline is more of a technical challenge.
One of the aspects I think will have more impact on the technical adoptability of the project is the use of conventions. Not all conventions are blockers, but some dictate directly how the tools are used.
So I suggest we categorize conventions in two ways:

• Hard conventions: conventions that are rigid, with not apparent workaround, and have an high impact on the way the format is used. These should be avoided.
• Soft conventions: conventions that do not impose themselves, and have an acceptable workaround. Should not be the primary choice, but if they bring value to the ecosystem, should be considered.

Sharing

It is likely that types will be shared across project, or even across studios so a consistent format that can be imported anywhere is important.

Tooling

• Provide automation focused command line tools to:
  • Validate files
  • Export to other formats, such as JSON
  • Conform files to a shareable standard format

Since this is a considerable work effort it is not realistic to expect all these features done at the same time, but since these same features will impact the development of the syntax itself, it is important to set them from the start.