Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synchronize two Playground instances #727

Merged
merged 86 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
f8104e3
Record and replay Playground SQL queries
adamziel Oct 28, 2023
4d9894c
Make MemFS events observable in the BasePHP class
adamziel Oct 28, 2023
23df8da
Extract journalMemfs as a simpler abstraction
adamziel Oct 29, 2023
99d4188
Remove unused impoerts
adamziel Oct 29, 2023
598dcd5
Sync proof of concept
adamziel Oct 29, 2023
1a9ced3
Two way sync
adamziel Oct 29, 2023
7b75f63
Deduplicate code
adamziel Oct 29, 2023
c4a19e5
Two way sync with an attempt to use disjoint numerical spaces for aut…
adamziel Oct 30, 2023
a01c8ea
Sync IDs in separate numerical spaces
adamziel Oct 30, 2023
49c5086
Tiny cleanup
adamziel Oct 30, 2023
92267f0
Add TODO to handle CREATE TABLE, ALTER TABLE, etc.
adamziel Oct 30, 2023
355cf66
Rewrite autoincrement values to match the peer
adamziel Oct 31, 2023
7ff9df7
Don't report replayed queries instead of relying on a counter
adamziel Oct 31, 2023
5d8438e
Get rid of runSqlQueries, simplify onChangeReceived
adamziel Oct 31, 2023
f5e5737
Bump autoincrement sequences when a table is created or altered
adamziel Oct 31, 2023
31eb208
Execute FS updates serially and draft the normalize() function to tur…
adamziel Oct 31, 2023
f5bd640
A failed attempt to implement normalize() by executing filesystem ope…
adamziel Nov 2, 2023
7fd082a
Prevent journaling of replayed FS changes
adamziel Nov 2, 2023
ba2c0cc
SQL Transactions support
adamziel Nov 2, 2023
9bad00f
Support for rollbacks
adamziel Nov 2, 2023
30f768b
Reformat
adamziel Nov 2, 2023
fa4b9d7
Document TODOs
adamziel Nov 2, 2023
279f044
Start experimenting with a trigger to assign ID values
adamziel Nov 2, 2023
b807945
Add event emitter capabilities to BasePHP
adamziel Nov 2, 2023
1decf3a
Explore different techniques of assigning IDs
adamziel Nov 3, 2023
a4883bf
Support for syncing insert queries with peer-generated autoincrement ids
adamziel Nov 3, 2023
f82047e
Only run the ID assignment trigger when not replaying queries from a …
adamziel Nov 3, 2023
d0bdbf3
Simplify the logic
adamziel Nov 3, 2023
27f721b
Add documentation, clean up the PHP side of the sync feature
adamziel Nov 3, 2023
2ac2087
Rollback uncommitted queries when the PHP request ends
adamziel Nov 3, 2023
c6d7adb
Clean up the code and put it in a separate "sync" directory
adamziel Nov 3, 2023
92a564e
Move sync to the new sync package
adamziel Nov 3, 2023
36ea97c
Clean up the package a little bit
adamziel Nov 3, 2023
b3e0905
Document what does "pk" mean
adamziel Nov 3, 2023
3552f44
Fix most TS type errors
adamziel Nov 3, 2023
839f4f6
Remove batching and debouncing
adamziel Nov 4, 2023
389276a
Debounce data packets
adamziel Nov 4, 2023
533ced4
Extract installSqlSyncMuPlugin as a separate function
adamziel Nov 4, 2023
41db536
Simplify batching transported changes
adamziel Nov 4, 2023
504df11
Add setupPlaygroundSync
adamziel Nov 4, 2023
6dc1a7c
Add a notion of a transport middleware to log messages
adamziel Nov 4, 2023
56f75e1
Replace middlewares with custom loggers
adamziel Nov 4, 2023
5cc299f
A small cleanup
adamziel Nov 4, 2023
19be3ac
Clean up the demo
adamziel Nov 4, 2023
652ec5d
Format, adjust config files
adamziel Nov 4, 2023
3f151a8
Use a proxy object for the FS journal
adamziel Nov 4, 2023
75e86be
Simplify and clean up the journaling bindings
adamziel Nov 4, 2023
8ff28fc
Remove NoopOperation
adamziel Nov 4, 2023
4800d1a
Remove a commented out code from base-php.ts
adamziel Nov 4, 2023
57b19b3
export interface BasePHP -> interface BasePHP
adamziel Nov 4, 2023
7d75425
Rewrite site URL on sync
adamziel Nov 4, 2023
4d6ffad
Formatting
adamziel Nov 4, 2023
b76ac65
Add 1 to Math.random() to ensure the result will always be above one …
adamziel Nov 4, 2023
73bb01d
Move pruning SQL queries to a middleware
adamziel Nov 4, 2023
9ff9b20
Remove the manual debouncedFlush() call
adamziel Nov 4, 2023
b7411cb
Cleanup
adamziel Nov 4, 2023
5c9e5c7
Simplify the transported data structure, only use a single level of n…
adamziel Nov 4, 2023
e17848c
Fix the setTimeout call
adamziel Nov 4, 2023
1fb7ec2
Cleanup the comments
adamziel Nov 4, 2023
83398ea
Unify the language to talk about journals and envelopes
adamziel Nov 4, 2023
f7da8ee
Fix CI
adamziel Nov 4, 2023
1f4c3d1
Promote journalingAllowed to a BasePHP property
adamziel Nov 4, 2023
ffae1b0
Add time traveling demo
adamziel Nov 4, 2023
2a8a846
Start drafting unit tests
adamziel Nov 4, 2023
ab6d127
Ignore the generated wp data files
adamziel Nov 4, 2023
4a49ce8
Move journaling to a new package and add unit tests
adamziel Nov 5, 2023
b09c2b9
Prototype normalizeFilesystemOperations
adamziel Nov 5, 2023
103165d
Implement normalizeFilesystemOperations()
adamziel Nov 5, 2023
4cf5315
Simplify normalizeFilesystemOperations slightly
adamziel Nov 5, 2023
9d36b30
Simplify normalizeFilesystemOperations further
adamziel Nov 5, 2023
c7d4577
Further simplify normalizeFilesystemOperations
adamziel Nov 5, 2023
7fb04dc
Simplify the data shape for TransportEnvelope to remove some dozen co…
adamziel Nov 5, 2023
55b7a80
Simplify basename implementation
adamziel Nov 5, 2023
2ac177b
Document paths.ts
adamziel Nov 5, 2023
caf0553
Replace the "atomic()" method with a synchronous replayFSJournal() pr…
adamziel Nov 5, 2023
680e7c9
Use maps and sets for event emitting
adamziel Nov 5, 2023
3809cf3
Declare host and headers directly instead of destructuring them
adamziel Nov 5, 2023
ce943cd
Use base64 instead of json_encode to pass strings to php
adamziel Nov 5, 2023
67cf982
Simplify phpVars() further to use just a single code branch
adamziel Nov 5, 2023
5908edd
Fix the time-traveling demo
adamziel Nov 6, 2023
2cff941
Use the correct SERVER_NAME to startup PHP
adamziel Nov 6, 2023
58946c4
Regenerate package-lock.json
adamziel Nov 6, 2023
63e4fa9
Test phpVar() by actually executing its output
adamziel Nov 6, 2023
f4eb63a
Lint
adamziel Nov 6, 2023
5cf097c
Move sync demos to the playground-website project
adamziel Nov 6, 2023
2c2e9ff
Unit test commits and rollbacks
adamziel Nov 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add documentation, clean up the PHP side of the sync feature
  • Loading branch information
adamziel committed Nov 3, 2023
commit 27f721b8dd35484a191d4bcc8bddd555eb860820
Original file line number Diff line number Diff line change
@@ -1,13 +1,5 @@
<?php

function playground_override_autoincrements_filter($query, $query_type)
{
if ($query_type === 'CREATE TABLE' || $query_type === 'ALTER TABLE') {
playground_override_autoincrement_algorithm();
}
}
add_filter('post_query_sqlite_db', 'playground_override_autoincrements_filter', -1000, 2);

/**
* Forces SQLite to use monotonically increasing values for every
* autoincrement column. For example, if sqlite_sequence says that
Expand Down Expand Up @@ -53,27 +45,30 @@ function playground_override_autoincrements_filter($query, $query_type)
* * Use INSTEAD OF triggers on a table – they only work with views
* * Read the entire row after INSERTing it, and reconstruct the query – too complex and be error-prone
* * Replace the ID column with a custom, non-autoincrement one – that's complex in SQLite plus it could mess up WP core db migrations
*
* @param int|null $local_id_offset The offset to use for the first AUTOINCREMENT value.
* @return void
*/
function playground_override_autoincrement_algorithm($local_id_offset = null)
function playground_sync_override_autoincrement_algorithm($local_id_offset = null)
{
if (null !== $local_id_offset) {
if (get_option('playground_id_offset')) {
// For now, the initial offset may only be set once.
// Changing it on the fly has no clear benefits, but
// it would be a pain to implement correctly and would
// introduce inconvenient gaps in the ID sequence.
throw new Exception(
"playground_sync_override_autoincrement_algorithm() was called twice with different " .
"values for \$local_id_offset. This is not supported."
);
}
// Store the default autoincrement offset for the current peer:
update_option('playground_id_offset', $local_id_offset);
}

playground_ensure_autoincrement_columns_view();

// Ensure the playground_sequence table exists:
$pdo = $GLOBALS['@pdo'];
$pdo->query("CREATE TABLE IF NOT EXISTS playground_sequence (
table_name varchar(255),
column_name varchar(255),
seq int default 0 not null,
PRIMARY KEY (table_name, column_name)
)");

// Insert all the AUTOINCREMENT table/column pairs that are not
// already tracked in playground_sequence:
$pdo = $GLOBALS['@pdo'];
$stmt = $pdo->prepare(<<<SQL
INSERT INTO playground_sequence
SELECT table_name, column_name, :seq FROM autoincrement_columns
Expand All @@ -83,7 +78,7 @@ function playground_override_autoincrement_algorithm($local_id_offset = null)
$stmt->execute([':seq' => get_option('playground_id_offset')]);

// Create any missing AFTER INSERT triggers:
foreach (playground_get_autoincrement_columns() as $table => $column) {
foreach (playground_sync_get_autoincrement_columns() as $table => $column) {
$pdo->query(<<<SQL
CREATE TRIGGER IF NOT EXISTS
force_seq_autoincrement_on_{$table}_{$column}
Expand All @@ -104,16 +99,87 @@ function playground_override_autoincrement_algorithm($local_id_offset = null)
}
}

function playground_get_autoincrement_columns()
/**
* Same as playground_sync_override_autoincrement_algorithm(), only runs after
* queries that modify the database schema, such as ALTER TABLE and CREATE TABLE.
*
* @param string $query MySQL Query
* @param string $query_type CREATE TABLE, ALTER TABLE, etc.
* @return void
*/
function playground_sync_override_autoincrement_on_newly_created_fields($query, $query_type)
{
if ($query_type === 'CREATE TABLE' || $query_type === 'ALTER TABLE') {
playground_sync_override_autoincrement_algorithm();
}
}

/**
* Ensures that $wpdb gets the actual ID assigned to the last inserted row
* as its $wpdb->insert_id value.
*
* The AFTER INSERT trigger overrides AUTOINCREMENT IDs provided by SQLite.
* However, the SQLite integration plugin uses the builtin last_insert_id()
* SQLite function which returns the original ID assigned by SQLite. That ID
* is no longer in the database, but there is no way to override it at the
* database level.
*
* We must, therefore, act at the application level. This function replaces
* the stale ID with the one assigned to the row by the AFTER INSERT trigger.
*
* @see playground_autoincrement_override_algorithm
*
* @param int $sqlite_last_insert_id The now-stale ID returned by last_insert_id().
* @param string $table_name The table name.
* @return int The ID actually stored in the last inserted row.
*/
function playground_sync_get_actual_last_insert_id($sqlite_last_insert_id, $table_name)
{
playground_ensure_autoincrement_columns_view();
$stmt = $GLOBALS['@pdo']->query('SELECT table_name, column_name FROM autoincrement_columns');
return $stmt ? $stmt->fetchAll(PDO::FETCH_KEY_PAIR) : [];
// Get the last relevant value from playground_sequence:
$stmt = $GLOBALS['@pdo']->prepare("SELECT * FROM playground_sequence WHERE table_name = :table_name");
$stmt->execute([':table_name' => $table_name]);
$result = $stmt->fetch(PDO::FETCH_ASSOC);
if ($result) {
return $result['seq'];
}
return $sqlite_last_insert_id;
}
/**
* Returns all auto-increment columns keyed by their table name.
*
* @return array A [$table => $column] array of all auto-increment columns.
*/
function playground_sync_get_autoincrement_columns()
{
return $GLOBALS['@pdo']
->query('SELECT table_name, column_name FROM autoincrement_columns')
->fetchAll(PDO::FETCH_KEY_PAIR);
}

function playground_ensure_autoincrement_columns_view()
/**
* Ensures that all the tables and views required by the synchronization
* process exist.
*
* This function may be called multiple times without causing an error.
*
* @return void
*/
function playground_sync_ensure_required_tables()
{
$GLOBALS['@pdo']->query(<<<SQL
$pdo = $GLOBALS['@pdo'];
/** @var PDO $pdo */
$pdo->query("CREATE TABLE IF NOT EXISTS playground_variables (
name TEXT PRIMARY KEY,
value TEXT
);");
$pdo->query("CREATE TABLE IF NOT EXISTS playground_sequence (
table_name varchar(255),
column_name varchar(255),
seq int default 0 not null,
PRIMARY KEY (table_name, column_name)
)");

$pdo->query(<<<SQL
CREATE VIEW IF NOT EXISTS autoincrement_columns AS
SELECT DISTINCT m.name as 'table_name', ti.name AS 'column_name', seq.seq AS 'seq'
FROM
Expand All @@ -132,65 +198,43 @@ function playground_ensure_autoincrement_columns_view()
}

/**
* Listens for SQL queries executed by WordPress and emits them to the JS side.
* Emits a SQL query to the JavaScript side of the Playground Sync
* feature.
*
* If the query is an INSERT and the local database implicitly assigned
* a primary key, this function will send the inserted rows instead of
* the original query. We do this because the original query doesn't
* give the remote peer enough information to reconstruct the row.
*
* @param string $query The SQL query to emit.
* @param string $query_type The type of the SQL query (e.g. SELECT, INSERT, UPDATE, DELETE).
* @param string $table_name The name of the table affected by the SQL query.
* @param array $insert_columns The columns affected by the INSERT query.
* @param int $last_insert_id The ID of the last inserted row (if applicable).
* @param int $affected_rows The number of affected rows.
* @return void
*/
function playground_sync_listen_for_sql_queries()
{
add_filter('sqlite_last_insert_id', function ($last_insert_id, $table_name) {
// Get last relevant value from playground_sequence
$pdo = $GLOBALS['@pdo'];
try {
$stmt = $pdo->prepare("SELECT * FROM playground_sequence WHERE table_name = :table_name");
$stmt->execute([':table_name' => $table_name]);
$result = $stmt->fetch(PDO::FETCH_ASSOC);
if ($result) {
return $result['seq'];
}
} catch (PDOException $e) {
// The `playground_sequence` table does not exists when this function runs for the first time.
// We can't create it yet because JS didn't assign us the AUTOINCREMENT ID offset yet.
// The only thing we can do is wait for the offset and ignore any PDO errors for now.
}
return $last_insert_id;
}, 0, 2);

// Report all SQL queries:
// Consider using SQLite's "update hook" instead of "sqlite_post_query" WordPress hook here:
add_filter('sqlite_post_query', 'playground_sync_emit_query', -1000, 6);

// Report all transaction-related queries:
$transaction_actions = [
'sqlite_begin_transaction' => 'START TRANSACTION',
'sqlite_commit' => 'COMMIT',
'sqlite_rollback' => 'ROLLBACK',
];
foreach ($transaction_actions as $action => $command) {
add_action($action, function ($success, $level) use ($command) {
if (0 !== $level) {
return;
}

post_message_to_js(json_encode([
'type' => 'sql',
'subtype' => 'transaction',
'success' => $success,
'command' => $command,
]));
}, 0, 2);
}
}

function playground_sync_emit_query($query, $query_type, $table_name, $insert_columns, $last_insert_id, $affected_rows)
function playground_sync_emit_mysql_query($query, $query_type, $table_name, $insert_columns, $last_insert_id, $affected_rows)
{
// Is it an INSERT that generated a new autoincrement value?
static $auto_increment_columns = null;
if ($auto_increment_columns === null) {
$auto_increment_columns = playground_get_autoincrement_columns();
$auto_increment_columns = playground_sync_get_autoincrement_columns();
}
$auto_increment_column = $auto_increment_columns[$table_name] ?? null;

$was_pk_generated = $query_type === 'INSERT' && $auto_increment_column && !in_array($auto_increment_column, $insert_columns, true);
if ($was_pk_generated) {
$rows = $GLOBALS['@pdo']->query("SELECT * FROM $table_name WHERE $auto_increment_column <= $last_insert_id ORDER BY $auto_increment_column DESC LIMIT $affected_rows")->fetchAll(PDO::FETCH_ASSOC);
// If so, get the inserted rows.
// It could be more than one, e.g. if the query was `INSERT INTO ... SELECT ...`.
$rows = $GLOBALS['@pdo']->query(<<<SQL
SELECT * FROM $table_name
WHERE $auto_increment_column <= $last_insert_id
ORDER BY $auto_increment_column DESC
LIMIT $affected_rows
SQL
)->fetchAll(PDO::FETCH_ASSOC);
// Finally, send each row to the JavaScript side.
foreach ($rows as $row) {
$row[$auto_increment_column] = (int) $row[$auto_increment_column];
post_message_to_js(json_encode([
Expand All @@ -206,6 +250,7 @@ function playground_sync_emit_query($query, $query_type, $table_name, $insert_co
return;
}

// Otherwise, simply send the query to the JavaScript side.
post_message_to_js(json_encode([
'type' => 'sql',
'subtype' => 'replay-query',
Expand All @@ -217,6 +262,38 @@ function playground_sync_emit_query($query, $query_type, $table_name, $insert_co
]));
}

/**
* Emits a transaction-related query to the JavaScript side of the
* Playground Sync.
*
* @param string $command The SQL statement (one of "START TRANSACTION", "COMMIT", "ROLLBACK").
* @param bool $success Whether the SQL statement was successful or not.
* @param int $nesting_level The nesting level of the transaction.
* @return void
*/
function playground_sync_emit_transaction_query($command, $success, $nesting_level)
{
// If we're in a nested transaction, SQLite won't really
// persist anything to the database. Let's ignore it and wait
// for the outermost transaction to finish.
if (0 !== $nesting_level) {
return;
}

post_message_to_js(json_encode([
'type' => 'sql',
'subtype' => 'transaction',
'success' => $success,
'command' => $command,
]));
}

/**
* Replays a list of SQL queries on a local database.
*
* @param array $queries An array of SQL queries to run.
* @return void
*/
function playground_sync_replay_queries($queries)
{
global $wpdb;
Expand All @@ -237,15 +314,15 @@ function playground_sync_replay_queries($queries)
$wpdb->query($query['query']);
}
} catch (PDOException $e) {
// Let's ignore errors related to UNIQUE constraints violation.
// Sometimes we'll ignore something we shouldn't, but for the most
// part, they are related to surface-level core mechanics like transients.
//
// Let's ignore errors related to UNIQUE constraints violation for now.
// They often relate to transient data that is not relevant to the
// synchronization process.
//
// This probably means we won't catch some legitimate issues.
// Let's keep an eye on this and see if we can eventually remove it.
// In the future, let's implement pattern matching on queries and
// prevent synchronizing things like transients.
var_dump("PDO Exception! " . $e->getMessage());
var_dump($e->getCode());
var_dump($query);
// prevent synchronizing transient data.

// SQLSTATE[23000]: Integrity constraint violation: 19 UNIQUE constraint failed
if ($e->getCode() === "23000") {
continue;
Expand All @@ -255,16 +332,34 @@ function playground_sync_replay_queries($queries)
}
}

// Don't override the AUTOINCREMENT IDs when replaying queries from
// another peer. The AFTER INSERT trigger will abstain from running
// when `is_replaying` is set to "yes".
$pdo = $GLOBALS['@pdo'];
$pdo->query("CREATE TABLE IF NOT EXISTS playground_variables (name TEXT PRIMARY KEY, value TEXT);");
$stmt = $pdo->prepare("INSERT OR REPLACE INTO playground_variables VALUES ('is_replaying', :is_replaying);");
$is_replaying = isset($GLOBALS['@REPLAYING_SQL']) && $GLOBALS['@REPLAYING_SQL'];
$stmt->execute([':is_replaying' => $is_replaying ? 'yes' : 'no']);
/**
* Sets up WordPress for a synchronized exchange of SQLite queries.
*
* @return void
*/
function playground_sync_start()
{
playground_sync_ensure_required_tables();

// Don't emit SQL queries we're just replaying from another peer.
if (!$is_replaying) {
playground_sync_listen_for_sql_queries();
// Don't override the AUTOINCREMENT IDs when replaying queries from
// another peer. The AFTER INSERT trigger will abstain from running
// when `is_replaying` is set to "yes".
$pdo = $GLOBALS['@pdo'];
$stmt = $pdo->prepare("INSERT OR REPLACE INTO playground_variables VALUES ('is_replaying', :is_replaying);");
$is_replaying = defined('REPLAYING_SQL') && REPLAYING_SQL;
$stmt->execute([':is_replaying' => $is_replaying ? 'yes' : 'no']);

// Don't emit SQL queries we're just replaying from another peer.
if (!$is_replaying) {
add_filter('sqlite_last_insert_id', 'playground_sync_get_actual_last_insert_id', 0, 2);

// Listens for SQL queries executed by WordPress and emit them to the JS side:
// @todo – consider using SQLite's "update hook" instead of "sqlite_post_query" WordPress hook here.
add_action('sqlite_translated_query_executed', 'playground_sync_emit_mysql_query', -1000, 6);
add_action('sqlite_transaction_query_executed', 'playground_sync_emit_transaction_query', -1000, 3);
}

add_filter('sqlite_translated_query_executed', 'playground_sync_override_autoincrement_on_newly_created_fields', -1000, 2);
}

playground_sync_start();
Loading