Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moves init schema to separate workflow + creates schema manager #3351

Merged
merged 11 commits into from
Mar 12, 2025

Conversation

alishakawaguchi
Copy link
Contributor

Creating the schema manager makes it easier to only update mysql to do incremental schema updates and let sqlserver and postgres stay the same

Copy link

linear bot commented Mar 11, 2025

Copy link

vercel bot commented Mar 11, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
neosync-docs ⬜️ Ignored (Inspect) Visit Preview Mar 12, 2025 10:01pm

@alishakawaguchi alishakawaguchi added the Devex Created by Linear-GitHub Sync label Mar 11, 2025
Copy link

codecov bot commented Mar 11, 2025

Codecov Report

Attention: Patch coverage is 42.12454% with 316 lines in your changes missing coverage. Please review.

Project coverage is 25.59%. Comparing base (a998809) to head (9f572d5).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/schema-manager/mssql/mssql.go 0.00% 104 Missing ⚠️
internal/schema-manager/postgres/postgres.go 55.46% 40 Missing and 13 partials ⚠️
...s/schemainit/activities/init-schema/init-schema.go 56.03% 36 Missing and 15 partials ⚠️
internal/schema-manager/mysql/mysql.go 34.66% 47 Missing and 2 partials ⚠️
worker/pkg/workflows/datasync/workflow/workflow.go 31.25% 21 Missing and 1 partial ⚠️
internal/schema-manager/schema-manager.go 52.63% 9 Missing ⚠️
worker/internal/cmds/worker/serve/serve.go 0.00% 8 Missing ⚠️
...rnal/schema-manager/not-supported/not-supported.go 0.00% 7 Missing ⚠️
...workflows/schemainit/workflow/register/register.go 0.00% 5 Missing ⚠️
...datasync/activities/sync-activity-opts/activity.go 40.00% 2 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3351      +/-   ##
==========================================
+ Coverage   25.51%   25.59%   +0.07%     
==========================================
  Files         401      409       +8     
  Lines       46905    47078     +173     
==========================================
+ Hits        11970    12048      +78     
- Misses      33772    33862      +90     
- Partials     1163     1168       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

github-actions bot commented Mar 11, 2025

Benchstat Geomean Results

0.72% sec/op, -0.03% B/op, 0.00% allocs/op

Benchstat results
                                   │ main-benchmark.txt │        branch-benchmark.txt        │
                                   │       sec/op       │    sec/op     vs base              │
CleanPostgresType-4                         42.63n ± 4%   41.34n ±  1%  -3.04% (p=0.002 n=6)
Runner_Single-4                             7.713µ ± 5%   7.846µ ±  2%       ~ (p=0.589 n=6)
GenerateBool-4                              11.45n ± 1%   11.35n ±  2%       ~ (p=0.058 n=6)
GenerateBusinessName-4                      285.4n ± 0%   309.9n ± 14%  +8.58% (p=0.002 n=6)
GenerateCardNumber-4                        28.41n ± 1%   28.54n ±  1%       ~ (p=0.240 n=6)
GenerateCategorical-4                       104.1n ± 2%   102.0n ±  0%  -2.07% (p=0.002 n=6)
GenerateCity-4                              173.7n ± 0%   173.9n ±  0%       ~ (p=1.000 n=6)
GenerateCountry-4                           105.2n ± 1%   106.3n ±  0%       ~ (p=0.216 n=6)
GenerateEmail-4                             1.346µ ± 0%   1.341µ ±  0%  -0.41% (p=0.026 n=6)
GenerateFirstName-4                         203.9n ± 0%   202.8n ±  3%       ~ (p=0.394 n=6)
GenerateFloat64-4                           28.73n ± 1%   28.69n ±  0%       ~ (p=0.087 n=6)
GenerateFullAddress-4                       1.429µ ± 3%   1.450µ ±  1%       ~ (p=0.121 n=6)
GenerateFullName-4                          1.222µ ± 1%   1.301µ ±  8%  +6.51% (p=0.002 n=6)
GenerateGender-4                            41.42n ± 0%   42.05n ±  1%  +1.50% (p=0.002 n=6)
GenerateInt64-4                             28.41n ± 2%   27.98n ±  1%  -1.50% (p=0.009 n=6)
GenerateInt64PhoneNumber-4                  51.46n ± 2%   51.64n ±  1%       ~ (p=0.368 n=6)
GenerateInternationalPhoneNumber-4          175.8n ± 1%   175.5n ±  0%       ~ (p=0.723 n=6)
GenerateIpAddress-4                         569.3n ± 0%   580.4n ±  1%  +1.97% (p=0.002 n=6)
GenerateLastName-4                          187.4n ± 1%   190.6n ±  2%  +1.71% (p=0.004 n=6)
GenerateRandomString-4                      985.6n ± 1%   981.9n ±  0%       ~ (p=0.169 n=6)
GenerateSHA256Hash-4                        1.016µ ± 0%   1.019µ ±  1%       ~ (p=0.485 n=6)
GenerateSSN-4                               305.8n ± 1%   302.2n ±  2%       ~ (p=0.139 n=6)
GenerateState-4                             105.2n ± 2%   105.5n ±  0%       ~ (p=0.509 n=6)
GenerateStreetAddress-4                     204.0n ± 1%   205.9n ±  2%  +0.98% (p=0.002 n=6)
GenerateStringPhoneNumber-4                 179.3n ± 0%   176.9n ±  0%  -1.37% (p=0.002 n=6)
GenerateUnixTimestamp-4                     77.13n ± 1%   77.26n ±  0%       ~ (p=0.310 n=6)
GenerateUsername-4                          351.5n ± 0%   357.2n ±  1%  +1.64% (p=0.002 n=6)
GenerateUTCTimestamp-4                      104.2n ± 1%   104.2n ±  2%       ~ (p=0.745 n=6)
GenerateUUID-4                              734.0n ± 0%   742.9n ±  0%  +1.22% (p=0.002 n=6)
GenerateZipcode-4                           105.2n ± 2%   105.7n ±  1%       ~ (p=0.195 n=6)
TransformCharacterScramble-4                171.8n ± 1%   172.2n ±  0%       ~ (p=0.333 n=6)
TransformE164PhoneNumber-4                  180.2n ± 3%   182.0n ±  0%       ~ (p=0.180 n=6)
TransformEmail-4                            1.798µ ± 2%   1.810µ ±  1%  +0.64% (p=0.035 n=6)
TransformFirstName-4                        233.9n ± 0%   236.7n ±  1%  +1.20% (p=0.002 n=6)
TransformFloat64-4                          110.8n ± 4%   116.1n ±  1%  +4.74% (p=0.002 n=6)
TransformFullName-4                         1.235µ ± 2%   1.255µ ±  1%  +1.66% (p=0.032 n=6)
TransformInt64-4                            33.74n ± 1%   33.25n ±  1%  -1.47% (p=0.009 n=6)
TransformInt64PhoneNumber-4                 57.84n ± 2%   58.09n ±  1%       ~ (p=0.071 n=6)
TransformLastName-4                         191.6n ± 0%   200.7n ±  1%  +4.78% (p=0.002 n=6)
TransformString-4                           1.022µ ± 0%   1.025µ ±  1%       ~ (p=0.208 n=6)
TransformStringPhoneNumber-4                211.0n ± 2%   209.6n ±  0%  -0.66% (p=0.002 n=6)
TransformUuid-4                             36.54n ± 1%   36.44n ±  1%  -0.29% (p=0.004 n=6)
geomean                                     194.0n        195.4n        +0.72%

                                   │ main-benchmark.txt │         branch-benchmark.txt         │
                                   │        B/op        │     B/op      vs base                │
CleanPostgresType-4                        0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                          2.312Ki ± 1%     2.285Ki ± 3%  -1.18% (p=0.026 n=6)
GenerateBool-4                             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                     304.0 ± 0%       304.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                       8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                      64.00 ± 0%       64.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                             160.0 ± 0%       160.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                            385.0 ± 0%       385.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                        208.0 ± 0%       208.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                          8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                      512.0 ± 0%       512.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                         418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                           16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                            7.000 ± 0%       7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                 8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4         40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                        31.00 ± 0%       31.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                         176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                     186.0 ± 0%       186.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                       288.0 ± 0%       288.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                              46.00 ± 0%       46.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                            24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                    240.0 ± 0%       240.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                    8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                         205.0 ± 0%       205.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                     24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                             80.00 ± 0%       80.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4               40.00 ± 3%       40.00 ± 3%       ~ (p=1.000 n=6)
TransformE164PhoneNumber-4                 40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                           489.0 ± 0%       489.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                       224.0 ± 0%       224.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                         88.00 ± 0%       88.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                        418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                           8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                        176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                          202.0 ± 0%       202.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4               56.00 ± 0%       56.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                            16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               ²                 -0.03%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                   │ main-benchmark.txt │        branch-benchmark.txt        │
                                   │     allocs/op      │ allocs/op   vs base                │
CleanPostgresType-4                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                            24.00 ± 0%     24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBool-4                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                     2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                       1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                      2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                             2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                          2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                            8.000 ± 0%     8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                          1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                      10.00 ± 0%     10.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                         6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                            0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                 1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                         2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                     3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                       7.000 ± 0%     7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                              3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                            2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                    2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                    1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                         6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                             3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                          2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4               2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformE164PhoneNumber-4                 3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                           14.00 ± 0%     14.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                       3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                        6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                          4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4               4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                            1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

Copy link
Member

@nickzelei nickzelei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left one comment regarding workflow id for child

siWf := &schemainit_workflow.Workflow{}
var wfResult schemainit_workflow.SchemaInitResponse
err := workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, workflow.ChildWorkflowOptions{
WorkflowID: workflow_shared.BuildChildWorkflowId(jobRunId, "init-schema", workflow.Now(ctx)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worth adding the destination id into the workflow id so that it is unique. yea we are using the time here but it might still cause dupes if it happens quick enough.

@alishakawaguchi alishakawaguchi merged commit 16998f7 into main Mar 12, 2025
19 checks passed
@alishakawaguchi alishakawaguchi deleted the alisha/neos-1774-mysql-schema-updates branch March 12, 2025 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Devex Created by Linear-GitHub Sync
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants