*.edgeql code generation

[colinhacks]

This is a proposal for a semi-standardized workflow for writing queries inside *.edgeql files and generating associated source files that provide a typesafe and idiomatic way to execute those queries.

*.edgeql files

Queries are written as plain EdgeQL inside *.edgeql files

// <project root>/path/to/myQuery.edgeql
select User {
  name, email
} filter .id = <uuid>$user_id

A generation command

An executable package/command should be provided to generate an executable function for each *.edgeql file.

This command should

1. Scan the project file tree for .edgeql files.
2. Read the contents
3. Retrieve a type/parameter descriptor from the database
4. Generate an appropriate source file.

Different language ecosystems provide different mechanisms for executing scripts of this nature, e.g. npx in the Node.js ecosystem, pipx in Python, etc.

• By default: a query file called myQuery.edgeql should produce a file in the same directory called myQuery.edgeql.<ext>
• This file should export a function, also called myQuery. In statically typed languages, this function should have an appropriate return type.
• The first argument should be a Client instance.
• The parameters to the query should be provided as named arguments where possible (e.g. in Python), or as an object passed as a second argument (e.g. in JS).

File names

The generated files should be easy to exclude with a single line in a .gitignore. This may involve a suffix or another naming convention.

• my_query.edgeql -> my_query_edgeql.py
• my_query.edgeql -> my_query.edgeql.ts

Detecting project root

The existence of an edgedb.toml is used to detecting the project root. Starting from the directory where the command is executed, the script should search up the file system for a directory containing edgedb.toml. In none is found, an error should be thrown.

Ignore migration files

Do not generate files associated with migration files. We'll need to respect the location of dbschema specified in edgedb.toml if/when it becomes possible to customize that (it isn't currently).

Connection args

The generation script should rely on the client library to resolve connection parameters, though it should also support the full set of command-line connection parameteters as an override:

• -h/--help
• --dsn
• --credentials-file
• -I/--instance
• -H/--host
• -P/--port
• -d/--database
• -u/--user
• --password
• --password-from-stdin
• --tls-ca-file
• --tls-security

--file

There should be a --file mode that will generate all "query functions" into a single file instead of generating per-query files. In the case of naming conflicts, an error should be thrown.

--watch

There should be a --watch mode that listens for changes to *.edgeql files. On the first run, each query will detect the project root, recursive find all *.edgeql files in that directory, and generate files appropriately. It should then start a file watcher that listens for changes to these files and the creation of any new *.edgeql files.

To avoid unnecessary regenerations, the generated files should contain a hash of the query as a comment on the first line.

// a83h984huifn48fjkf

export function myQuery(){ ... }

This hash can be read and used to determine if a given query has changed and needs to be re-generated.

--target

Where necessary, a --target flag should be provided to provide support for different language variants. For instance, JavaScript vs TypeScript.

Worked example

Query file getUser.edgeql

// ./path/to/getUser.edgeql
select User {
  name, email
} filter .id = <uuid>$user_id

The user executes an npx command to generate appropriate files

npx @edgedb/generate edgeql-files --target ts

A getUser.edgeql.ts file is generated with the following contents

// <hash here>
import type {Executor} from "edgedb";

export function getUser(client: Executor, params: {id: string}): {name: string; email: string;} {
  // this will be either query, querySingle, queryRequired, or queryRequiredSingle depending on the introspected result cardinality
  return client.querySingle<getUser>(`<query here>`, params);
}

The generated query is consumed like so

import {createClient} from "edgedb";
import {getUser} from "./path/to/getUser";

const client = createClient();
await getUser(client, {id: "abc..."});
// { name: "Jules", ... }

Comparison to a full query builder

Pros

• Users can write plain EdgeQL, which is less verbose than a query builder, especially for shapes and operators
• Users don't need to learn the equivalent query builder syntax for all EdgeQL operations
• Massive implementational simplification compared to a query builder
• Many users are familiar with this workflow from GraphQL

Cons

• Code generation
• No autocomplete or type checking in *.edgeql files until LSP lands
• Your queries are siloed in *.edgeql files (this is also kinda nice in some sense)

[fmoor]

This hash can be read and used to determine if a given query has changed and needs to be re-generated.

This doesn't account for changes in the edgedb client API. If a user does a major version upgrade then the files won't be updated and may not compile/run.

[colinhacks]

Adding the following:

The hash should only be checked when file changes are detected in --watch mode. All files should be regenerated when executing the query for the first time, in case the version of the generator or underlying database has changed in the meantime.

[1st1]

To avoid unnecessary regenerations, the generated files should contain a hash of the query as a comment on the first line.

Potentially the hash should also include the latest applied migration hash.

[nsidnev]

I have a question about the obligation to use an executable script to generate code. Does it really need to be a command that has to be run manually?

For example in the case of Elixir I created edgedb_ecto which does similar things to the above (except automatic options recognizing) and generates modules and functions when the Elixir application is being compiled. Because of this, I don't need, for example, the --watch flag and a separate daemon to watch the files, because any code watcher (such as the one used with phoenix) will run automatic regeneration of queries + application recompilation when changes are made.

I also have 2 suggestions about generators usage:

1. By default: a query file called myQuery.edgeql should produce a file in the same directory called myQuery.{extension}

   I think there should be a --output-path flag or something similar, because IMO users don't always want to have generated code next to their queries.

2. Can't we rely on the CLI to run generators instead of running (and remembering) separate executables for each language manually? Something like protoc with it plugins does. It could help with a several things:

   1. It seems to me that this way generators wouldn't need to implement all logic of option resolution and they could get prepared options at the start.
   2. In case user uses several languages, I think that typing, for example, edgedb generate --lang python --watch with shell completion would be more convenient than running pipx in one directory, npx in another and so on.

[tailhook]

To avoid unnecessary regenerations, the generated files should contain a hash of the query as a comment on the first line.

I think we might want to add // GENERATED DO NOT EDIT or something like that at the start of the file, so hash line should be marked, like:

// Sha256hash: a83h984huifn48fjkf

Either that hash should include or another field should contain version of the tool. So you don't have to run clean/force schenanigans after an update.

[fmoor]

// Sha256hash: a83h984huifn48fjkf

maybe a more specific prefix? //edgeql:xxxxxxxxxxxxxxxxxxx?

[tailhook]

Is there a reason of why you export getUser instead of export default (in non---file mode)?

What happens with folders in --file mode? Are they omitted? Or do they create extra objects/namespaces in the module?

[fmoor]

What happens with folders in --file mode?

Perhaps all queries in a particular directory get grouped into one output file?

[tailhook]

though it should also support the full set of command-line connection parameteters as an override

The problem with this is not just that one have to implement support for all of the arguments, but also that we can add more later and everyone should follow. Do we have a strong use case for that? I think environment configuration is good enough and simplifies implementation and maintenance.

[fmoor]

I agree. In go the connection options make even less sense because the command will be committed to source and run with go generate so then everyone using the project would need to coerce their instance to listen on --port 1234 or something.