Skip to content

Commit

Permalink
added build stuff from electric-sql
Browse files Browse the repository at this point in the history
  • Loading branch information
lacanoid committed Jun 6, 2023
1 parent cad2962 commit 26558a2
Show file tree
Hide file tree
Showing 6 changed files with 83 additions and 3 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
*~
/results/
/*--*.sql
electric-ddlx-*.sql
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ but not conststraints, indexes, triggers etc.
Use `ddlx_alter()` to create those after the data is loaded.
- more use of explicit casts
- `ddlx_create_trigger()` improvements by [PegoraroF10](https://github.com/PegoraroF10), now with replica and always triggers
- change `sysid` in some places to `oid`

Version 0.22
------------
Expand Down
10 changes: 10 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,13 @@ $(DATA_built): ddlx.sql
@echo "Building extension version" $(EXT_VERSION) "for Postgres version" $(VERSION)
VERSION=${VERSION} ./bin/pgsqlpp $^ >$@


.PHONY: electric
electric: ddlx.sql
@echo "Generating Electric extension SQL for Postgres version" $(VERSION)
$(eval tmpfile := $(shell mktemp --suffix=.sql))
$(eval outfile = electric-ddlx-${VERSION}.sql)
VERSION=${VERSION} ./bin/pgsqlpp $^ > $(tmpfile)
elixir ./bin/electric.exs --in ${tmpfile} --out $(outfile)
@echo "Extension file written to " $(outfile)

48 changes: 48 additions & 0 deletions README-electric.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Building for Electric

Electric maintains a fork of the original because we need a few tweaks
on the upstream behaviour, specifically:

- Set column defaults within the `CREATE TABLE (...)` block, rather than via
`ALTER TABLE .. SET DEFAULT` statements after the table creation.

- Define table constraints within the create table block too

Both of these are to support direct PostgreSQL -> SQLite command translation.
For example, SQLite does not support `ALTER TABLE .. SET DEFAULT` and we are
currently very strict about the migrations that we allow on electrified tables.

So it's best for our use case that tables are created in one shot, without
additional `ALTER TABLE` commands.

As PG versions progress it may be necessary to update these functions, hence
this fork which applies our patches in the `ddlx.sql` file which has
pre-processor statements for multiple PG version support.

## Generating Electric extension SQL

For the purposes of Electric, "extension" does not mean a postgresql extension.
Electric has its own extension mechanism based on simple SQL statements,
maintained via a basic migration system.

That extension mechanism expects everything to be written within the `electric`
pg schema. PostgreSQL allows for extensions to be relocated into any schema via
the built-in extension mechanism. Since we can't use that, we need to be able
to write our functions into a specific schema.

So we have a pre-processor script that prefixes all function definitions and
calls with a `@schemaname@` prefix which we can replace when applying the sql
to the live pg database.

To generate a new "electrified" ddlx script, run the following:

```
make electric VERSION=$PG_VERSION
```

This will write a processed file to `electric-ddlx-$PG_VERSION.sql` in this
repo.

This file should be moved into the electric repo somewhere under
`components/electric/lib/electric/postgres/extension/migrations/$TIMESTAMP/ddlx-$PG_VERSION.sql`.
See the existing ddlx migration version for a template.
20 changes: 20 additions & 0 deletions bin/electric.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{opts, _, _} = OptionParser.parse(System.argv(), strict: [in: :string, out: :string])
source = File.read!(opts[:in])
dest = Keyword.fetch!(opts, :out)

IO.puts("Preprocessing #{opts[:in]} and writing to #{dest}")

migration =
Regex.scan(~r/^CREATE OR REPLACE FUNCTION ([a-z_]+\()/m, source)
|> Enum.map(fn [_, f] -> f end)
|> Enum.uniq()
|> Enum.reduce(source, fn func, source ->
String.replace(
source,
func,
"@schemaname@.#{func}",
global: true
)
end)

File.write!(dest, migration, [])
6 changes: 3 additions & 3 deletions ddlx.sql
Original file line number Diff line number Diff line change
Expand Up @@ -528,7 +528,7 @@ CREATE OR REPLACE FUNCTION ddlx_get_constraints(
OUT is_deferrable boolean,
OUT initially_deferred boolean,
OUT regclass oid,
OUT sysid oid,
OUT oid oid,
OUT is_local boolean)
RETURNS SETOF record LANGUAGE sql AS $function$
SELECT nc.nspname AS namespace,
Expand Down Expand Up @@ -668,13 +668,13 @@ $function$;

CREATE OR REPLACE FUNCTION ddlx_get_functions(
regproc default null,
OUT sysid oid, OUT namespace name, OUT name name, OUT comment text,
OUT oid oid, OUT namespace name, OUT name name, OUT comment text,
OUT owner name, OUT sql_identifier text, OUT language name, OUT attributes text,
OUT retset boolean, OUT is_trigger boolean, OUT returns text, OUT arguments text,
OUT definition text, OUT security text, OUT is_strict text, OUT argtypes oidvector,
OUT cost real, OUT rows real)
RETURNS SETOF record LANGUAGE sql AS $function$
SELECT p.oid AS sysid,
SELECT p.oid AS oid,
s.nspname AS namespace,
p.proname AS name,
pg_description.description AS comment,
Expand Down

0 comments on commit 26558a2

Please sign in to comment.