Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runconfigs V2 - adds stability, removes complexity #3285

Merged
merged 40 commits into from
Mar 6, 2025

Conversation

alishakawaguchi
Copy link
Contributor

@alishakawaguchi alishakawaguchi commented Feb 20, 2025

Simplifies run config and query builder logic.

For circular dependencies and subsets removes complicated logic around optimal path. Now it determines update configs based on tables nullable foreign keys. A separate update config is created for each nullable foreign key on the table.

Moves subset joining logic into run config builder because we create update configs for subset tables now. It computes the shortest path from every child table to all parent subsets. This reduces unnecessary and complicated joins

@alishakawaguchi alishakawaguchi self-assigned this Feb 20, 2025
Copy link

vercel bot commented Feb 20, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
neosync-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 6, 2025 5:08pm

Copy link

github-actions bot commented Feb 20, 2025

Benchstat Geomean Results

0.41% sec/op, -0.08% B/op, 0.00% allocs/op

Benchstat results
                                   │ main-benchmark.txt │        branch-benchmark.txt         │
                                   │       sec/op       │    sec/op     vs base               │
CleanPostgresType-4                         42.63n ± 4%   42.97n ± 90%        ~ (p=0.121 n=6)
Runner_Single-4                             7.713µ ± 5%   7.683µ ± 17%        ~ (p=0.485 n=6)
GenerateBool-4                              11.45n ± 1%   11.32n ±  1%   -1.09% (p=0.015 n=6)
GenerateBusinessName-4                      285.4n ± 0%   299.8n ±  3%   +5.03% (p=0.002 n=6)
GenerateCardNumber-4                        28.41n ± 1%   28.36n ±  2%        ~ (p=0.463 n=6)
GenerateCategorical-4                       104.1n ± 2%   102.4n ±  0%   -1.68% (p=0.002 n=6)
GenerateCity-4                              173.7n ± 0%   173.2n ±  1%        ~ (p=0.058 n=6)
GenerateCountry-4                           105.2n ± 1%   105.6n ±  1%        ~ (p=0.457 n=6)
GenerateEmail-4                             1.346µ ± 0%   1.336µ ±  0%   -0.74% (p=0.002 n=6)
GenerateFirstName-4                         203.9n ± 0%   203.2n ±  0%   -0.32% (p=0.039 n=6)
GenerateFloat64-4                           28.73n ± 1%   28.62n ±  0%   -0.40% (p=0.048 n=6)
GenerateFullAddress-4                       1.429µ ± 3%   1.442µ ±  2%        ~ (p=0.162 n=6)
GenerateFullName-4                          1.222µ ± 1%   1.243µ ±  1%   +1.76% (p=0.002 n=6)
GenerateGender-4                            41.42n ± 0%   46.86n ±  0%  +13.12% (p=0.002 n=6)
GenerateInt64-4                             28.41n ± 2%   27.86n ±  1%   -1.94% (p=0.002 n=6)
GenerateInt64PhoneNumber-4                  51.46n ± 2%   51.54n ±  0%        ~ (p=0.818 n=6)
GenerateInternationalPhoneNumber-4          175.8n ± 1%   176.9n ±  1%   +0.60% (p=0.011 n=6)
GenerateIpAddress-4                         569.3n ± 0%   577.8n ±  1%   +1.49% (p=0.002 n=6)
GenerateLastName-4                          187.4n ± 1%   187.4n ±  1%        ~ (p=0.922 n=6)
GenerateRandomString-4                      985.6n ± 1%   978.8n ±  3%        ~ (p=0.180 n=6)
GenerateSHA256Hash-4                        1.016µ ± 0%   1.009µ ±  0%   -0.74% (p=0.004 n=6)
GenerateSSN-4                               305.8n ± 1%   304.6n ±  0%   -0.41% (p=0.002 n=6)
GenerateState-4                             105.2n ± 2%   104.8n ±  0%        ~ (p=0.113 n=6)
GenerateStreetAddress-4                     204.0n ± 1%   205.9n ±  0%   +0.98% (p=0.002 n=6)
GenerateStringPhoneNumber-4                 179.3n ± 0%   171.9n ±  1%   -4.16% (p=0.002 n=6)
GenerateUnixTimestamp-4                     77.13n ± 1%   77.14n ±  0%        ~ (p=1.000 n=6)
GenerateUsername-4                          351.5n ± 0%   354.4n ±  0%   +0.84% (p=0.002 n=6)
GenerateUTCTimestamp-4                      104.2n ± 1%   103.9n ±  0%   -0.29% (p=0.048 n=6)
GenerateUUID-4                              734.0n ± 0%   741.5n ±  1%   +1.03% (p=0.002 n=6)
GenerateZipcode-4                           105.2n ± 2%   106.5n ±  0%        ~ (p=0.058 n=6)
TransformCharacterScramble-4                171.8n ± 1%   172.8n ±  0%   +0.64% (p=0.043 n=6)
TransformE164PhoneNumber-4                  180.2n ± 3%   179.7n ±  1%        ~ (p=0.290 n=6)
TransformEmail-4                            1.798µ ± 2%   1.790µ ±  1%   -0.47% (p=0.013 n=6)
TransformFirstName-4                        233.9n ± 0%   240.4n ±  0%   +2.78% (p=0.002 n=6)
TransformFloat64-4                          110.8n ± 4%   115.3n ±  1%   +4.01% (p=0.022 n=6)
TransformFullName-4                         1.235µ ± 2%   1.253µ ±  1%   +1.50% (p=0.035 n=6)
TransformInt64-4                            33.74n ± 1%   33.12n ±  1%   -1.82% (p=0.002 n=6)
TransformInt64PhoneNumber-4                 57.84n ± 2%   58.03n ±  1%        ~ (p=0.058 n=6)
TransformLastName-4                         191.6n ± 0%   191.5n ±  0%        ~ (p=0.383 n=6)
TransformString-4                           1.022µ ± 0%   1.022µ ±  3%        ~ (p=0.978 n=6)
TransformStringPhoneNumber-4                211.0n ± 2%   204.9n ±  0%   -2.87% (p=0.002 n=6)
TransformUuid-4                             36.54n ± 1%   36.59n ±  1%        ~ (p=0.515 n=6)
geomean                                     194.0n        194.8n         +0.41%

                                   │ main-benchmark.txt │         branch-benchmark.txt         │
                                   │        B/op        │     B/op      vs base                │
CleanPostgresType-4                        0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                          2.312Ki ± 1%     2.290Ki ± 1%       ~ (p=0.102 n=6)
GenerateBool-4                             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                     304.0 ± 0%       304.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                       8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                      64.00 ± 0%       64.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                             160.0 ± 0%       160.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                            385.0 ± 0%       385.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                        208.0 ± 0%       208.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                          8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                      512.0 ± 0%       512.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                         418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                           16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                            7.000 ± 0%       7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                 8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4         40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                        31.00 ± 0%       31.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                         176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                     186.0 ± 0%       186.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                       288.0 ± 0%       288.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                              46.00 ± 0%       46.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                            24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                    240.0 ± 0%       240.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                    8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                         205.0 ± 0%       205.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                     24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                             80.00 ± 0%       80.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4               40.00 ± 3%       39.00 ± 3%       ~ (p=0.567 n=6)
TransformE164PhoneNumber-4                 40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                           489.0 ± 0%       489.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                       224.0 ± 0%       224.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                         88.00 ± 0%       88.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                        418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                           8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                        176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                          202.0 ± 0%       202.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4               56.00 ± 0%       56.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                            16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               ²                 -0.08%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                   │ main-benchmark.txt │        branch-benchmark.txt        │
                                   │     allocs/op      │ allocs/op   vs base                │
CleanPostgresType-4                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                            24.00 ± 0%     24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBool-4                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                     2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                       1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                      2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                             2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                          2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                            8.000 ± 0%     8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                          1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                      10.00 ± 0%     10.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                         6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                            0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                 1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                         2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                     3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                       7.000 ± 0%     7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                              3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                            2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                    2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                    1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                         6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                             3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                          2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4               2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformE164PhoneNumber-4                 3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                           14.00 ± 0%     14.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                       3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                        6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                          4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4               4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                            1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

Copy link

codecov bot commented Feb 26, 2025

Codecov Report

Attention: Patch coverage is 54.35502% with 414 lines in your changes missing coverage. Please review.

Project coverage is 24.66%. Comparing base (6c923eb) to head (19f822a).
Report is 12 commits behind head on main.

Files with missing lines Patch % Lines
worker/pkg/select-query-builder/querybuilder.go 0.00% 269 Missing ⚠️
internal/runconfigs/runconfigs.go 59.01% 72 Missing and 3 partials ⚠️
internal/benthos/benthos-builder/builders/sql.go 11.76% 15 Missing ⚠️
worker/pkg/select-query-builder/subset.go 0.00% 15 Missing ⚠️
internal/runconfigs/builder.go 96.82% 7 Missing and 3 partials ⚠️
...ernal/benthos/benthos-builder/builders/dynamodb.go 0.00% 9 Missing ⚠️
...ternal/benthos/benthos-builder/builders/mongodb.go 0.00% 9 Missing ⚠️
...ternal/benthos/benthos-builder/generate-benthos.go 0.00% 2 Missing ⚠️
cli/internal/cmds/neosync/sync/util.go 50.00% 1 Missing ⚠️
...nternal/benthos/benthos-builder/builders/aws-s3.go 0.00% 1 Missing ⚠️
... and 8 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3285      +/-   ##
==========================================
+ Coverage   24.61%   24.66%   +0.04%     
==========================================
  Files         394      395       +1     
  Lines       45997    45867     -130     
==========================================
- Hits        11323    11311      -12     
+ Misses      33544    33435     -109     
+ Partials     1130     1121       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alishakawaguchi alishakawaguchi added the Devex Created by Linear-GitHub Sync label Mar 1, 2025
@alishakawaguchi alishakawaguchi changed the title WIP - runconfigs v2 Runconfigs V2 - adds stability, removes complexity Mar 4, 2025
qb := NewSelectQueryBuilder("public", driver, subsetByForeignKeyConstraints, pageLimit)
querymap := map[string]*sqlmanager_shared.SelectQuery{}
for _, cfg := range runConfigs {
query, _, pageQuery, isNotForeignKeySafe, err := qb.BuildQuery(cfg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confused as to where the isNotForeignKeySafe is defined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should probably rename that. It really just tells me when a query is a subset query and needs to turn skip fk violations when inserting or updating. It gets determined when building the select query.

@alishakawaguchi alishakawaguchi merged commit cd0164e into main Mar 6, 2025
19 checks passed
@alishakawaguchi alishakawaguchi deleted the alisha/runconfigs-v2 branch March 6, 2025 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Devex Created by Linear-GitHub Sync
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants