feat: improve buf rate limit (backport #4133) #4206

mergify · 2024-06-21T12:21:28Z

Description

Remove third_party_paths from the chain config.
Decrease the Buf. Build requests by grouping the requests into only one using the --path for the generate command.
Create an option to run the proto files one by one so we can skip the workspace for the third-party modules, like the swagger file generation.
Create a cache for the buf file, increasing the Ignite run speed.
This is an automatic backport of pull request feat: improve buf rate limit #4133 done by Mergify.

* draft improvements * use --path flag mutilple times * remove `Build.Proto.ThirdPartyPaths` from the config * add logic for run buf by folder or by file * update cosmos-sdk * fix exclude files for swagger * fix buf cache * add changelog * update buf deps * fix unit test * fix wrong proto go path * clear buf cache * fix doctor tests and remove `third_party_paths` from the docs * improve Test_extractRootModulePath test cases * Update ignite/pkg/cosmosbuf/cache.go Co-authored-by: Jerónimo Albi <[email protected]> * fix changelog.md * improve cache logic * create a dircache pkg * add comments * rename methods `CopyCache` to `CopyTo` and `SaveCache` to `Save` --------- Co-authored-by: Pantani <Pantani> Co-authored-by: Jerónimo Albi <[email protected]> (cherry picked from commit 0b41262) # Conflicts: # ignite/cmd/cmd.go # ignite/config/chain/base/config.go # ignite/config/chain/v1/testdata/config.yaml # ignite/config/chain/v1/testdata/config2.yaml # ignite/internal/analytics/analytics.go # ignite/pkg/cosmosbuf/buf.go # ignite/pkg/cosmosgen/generate_go.go # ignite/pkg/cosmosgen/generate_openapi.go # integration/doctor/testdata/config-need-migrate.txt

mergify · 2024-06-21T12:21:29Z

Cherry-pick of 0b41262 has failed:

On branch mergify/bp/release/v28.x.y/pr-4133
Your branch is up to date with 'origin/release/v28.x.y'.

You are currently cherry-picking commit 0b412628.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   changelog.md
	modified:   docs/docs/02-guide/06-ibc.md
	modified:   docs/docs/02-guide/07-interchange/02-init.md
	modified:   docs/docs/08-references/02-config.md
	modified:   docs/versioned_docs/version-v0.25/kb/03-config.md
	modified:   docs/versioned_docs/version-v0.25/kb/06-proto.md
	modified:   ignite/config/chain/v1/config_test.go
	deleted:    ignite/pkg/cosmosbuf/buf_test.go
	modified:   ignite/pkg/cosmosgen/cosmosgen.go
	modified:   ignite/pkg/cosmosgen/generate.go
	new file:   ignite/pkg/cosmosgen/generate_openapi_test.go
	modified:   ignite/pkg/cosmosgen/generate_typescript.go
	new file:   ignite/pkg/dircache/cache.go
	new file:   ignite/pkg/dircache/cache_test.go
	new file:   ignite/pkg/dircache/testdata/subdata/subfile
	new file:   ignite/pkg/dircache/testdata/testfile
	modified:   ignite/pkg/xos/files.go
	modified:   ignite/pkg/xos/files_test.go
	modified:   ignite/services/chain/generate.go
	modified:   ignite/services/scaffolder/scaffolder.go
	modified:   ignite/templates/app/files/proto/buf.lock

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   ignite/cmd/cmd.go
	both modified:   ignite/config/chain/base/config.go
	both modified:   ignite/config/chain/v1/testdata/config.yaml
	both modified:   ignite/config/chain/v1/testdata/config2.yaml
	both modified:   ignite/internal/analytics/analytics.go
	both modified:   ignite/pkg/cosmosbuf/buf.go
	both modified:   ignite/pkg/cosmosgen/generate_go.go
	both modified:   ignite/pkg/cosmosgen/generate_openapi.go
	both modified:   integration/doctor/testdata/config-need-migrate.txt

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

github-actions · 2024-06-21T12:23:33Z

Visit the preview URL for this PR (updated for commit ba2840e):

https://igntservices-docs--pr4206-mergify-bp-release-v-g38uu843.web.app

_{(expires Fri, 12 Jul 2024 16:22:20 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 95379efd94dd497aaa37c2d0354e6e2cafca5ec5}

codecov · 2024-07-04T19:19:47Z

Codecov Report

Attention: Patch coverage is 8.27338% with 255 lines in your changes missing coverage. Please review.

Project coverage is 25.94%. Comparing base (232b10d) to head (cdd543f).
Report is 37 commits behind head on release/v28.x.y.

❗ Current head cdd543f differs from pull request most recent head ba2840e

Please upload reports for the commit ba2840e to get more accurate results.

Additional details and impacted files

@@                 Coverage Diff                 @@
##           release/v28.x.y    #4206      +/-   ##
===================================================
+ Coverage            24.74%   25.94%   +1.19%     
===================================================
  Files                  297      301       +4     
  Lines                24979    25653     +674     
===================================================
+ Hits                  6182     6655     +473     
- Misses               18230    18417     +187     
- Partials               567      581      +14

Files	Coverage Δ
ignite/pkg/cosmosanalysis/app/app.go	`58.00% <ø> (ø)`
ignite/pkg/cosmosanalysis/cosmosanalysis.go	`84.02% <ø> (ø)`
ignite/pkg/cosmosgen/generate_vuex.go	`0.00% <ø> (ø)`
ignite/pkg/repoversion/repoversion.go	`57.47% <ø> (ø)`
ignite/services/chain/init.go	`0.00% <ø> (ø)`
ignite/services/scaffolder/scaffolder.go	`0.00% <ø> (ø)`
ignite/pkg/cosmosgen/generate_typescript.go	`0.00% <0.00%> (ø)`
ignite/services/chain/generate.go	`0.00% <0.00%> (ø)`
ignite/cmd/cmd.go	`0.00% <0.00%> (ø)`
ignite/pkg/cosmosgen/cosmosgen.go	`3.33% <0.00%> (+0.15%)`	⬆️
... and 6 more

... and 2 files with indirect coverage changes

# Conflicts: # ignite/pkg/cosmosbuf/buf.go

julienrbrt

tACK!

julienrbrt

Now #4202 is broken with a different message btw and on non-minimal chains as well.

error while running command /home/julien/tools/go/bin/buf generate /home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1 --template=/home/julien/downloads/foo2/proto/buf.gen.sta.yaml --output=/tmp/gen-openapi-module-spec2507011614 --error-format=json --log-format=json --path=/home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1/tx.proto --path=/home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1/vesting.proto: Failure: failed to build input "proto/cosmos/vesting/v1beta1" because it is contained by module at path "proto" specified in your configuration, you must provide the workspace or module as the input, and filter to this path using --path : exit status 1

Pantani · 2024-07-10T01:07:08Z

Now #4202 is broken with a different message btw and on non-minimal chains as well.

error while running command /home/julien/tools/go/bin/buf generate /home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1 --template=/home/julien/downloads/foo2/proto/buf.gen.sta.yaml --output=/tmp/gen-openapi-module-spec2507011614 --error-format=json --log-format=json --path=/home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1/tx.proto --path=/home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1/vesting.proto: Failure: failed to build input "proto/cosmos/vesting/v1beta1" because it is contained by module at path "proto" specified in your configuration, you must provide the workspace or module as the input, and filter to this path using --path : exit status 1

@julienrbrt, we need to merge the other PRs after this one to the ts client generation works again:

#4226
#4225
#4224

Pantani · 2024-07-10T01:40:31Z

Now #4202 is broken with a different message btw and on non-minimal chains as well.
error while running command /home/julien/tools/go/bin/buf generate /home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1 --template=/home/julien/downloads/foo2/proto/buf.gen.sta.yaml --output=/tmp/gen-openapi-module-spec2507011614 --error-format=json --log-format=json --path=/home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1/tx.proto --path=/home/julien/tools/go/pkg/mod/github.com/cosmos/[email protected]/proto/cosmos/vesting/v1beta1/vesting.proto: Failure: failed to build input "proto/cosmos/vesting/v1beta1" because it is contained by module at path "proto" specified in your configuration, you must provide the workspace or module as the input, and filter to this path using --path : exit status 1

@julienrbrt, we need to merge the other PRs after this one to the ts client generation works again:

#4226 #4225 #4224

but we also need to merge this one first for the others works, because the backport to update the buf version is already merged

mergify bot added the conflicts label Jun 21, 2024

mergify bot requested review from ilgooz, salmad3, toschdev, jeronimoalbi, Pantani and julienrbrt as code owners June 21, 2024 12:21

mergify bot mentioned this pull request Jun 21, 2024

feat: improve buf rate limit #4133

Merged

github-actions bot added component:docs Documentation additions or improvements. component:ci CI/CD workflow and automated jobs. component:templates component:configs component:cmd type:services Service-related issues. type:internal component:packages labels Jun 21, 2024

julienrbrt self-assigned this Jun 21, 2024

julienrbrt marked this pull request as draft June 21, 2024 12:22

Pantani approved these changes Jun 27, 2024

View reviewed changes

Pantani closed this Jul 4, 2024

Pantani deleted the mergify/bp/release/v28.x.y/pr-4133 branch July 4, 2024 13:46

Pantani added backport/v28.x.y Backport to v28.x.y and removed backport/v28.x.y Backport to v28.x.y labels Jul 4, 2024

Pantani restored the mergify/bp/release/v28.x.y/pr-4133 branch July 4, 2024 17:52

Pantani reopened this Jul 4, 2024

fix conflicts

63ae4ec

Pantani force-pushed the mergify/bp/release/v28.x.y/pr-4133 branch from 99a1372 to 63ae4ec Compare July 4, 2024 18:17

Pantani added 4 commits July 4, 2024 20:18

Merge branch 'release/v28.x.y' into mergify/bp/release/v28.x.y/pr-4133

549b115

rollback unused changes

e341442

rollback docs

166fef0

fix changelog

c40ba37

github-actions bot removed the component:docs Documentation additions or improvements. label Jul 4, 2024

Pantani marked this pull request as ready for review July 4, 2024 18:31

Pantani added 3 commits July 4, 2024 20:33

add missing variable

2cfe42c

fix buf dependencies

b91aae0

update the chain deps

cdd543f

Pantani added 3 commits July 5, 2024 02:56

fix operationid for the swagger combine

d516178

Merge branch 'release/v28.x.y' into mergify/bp/release/v28.x.y/pr-4133

167273e

# Conflicts: # ignite/pkg/cosmosbuf/buf.go

rollback ThirdPartyPaths delete

ba2840e

github-actions bot added the component:docs Documentation additions or improvements. label Jul 5, 2024

julienrbrt approved these changes Jul 9, 2024

View reviewed changes

julienrbrt requested changes Jul 9, 2024

View reviewed changes

julienrbrt removed the conflicts label Jul 9, 2024

Pantani requested a review from julienrbrt July 10, 2024 01:07

julienrbrt approved these changes Jul 10, 2024

View reviewed changes

julienrbrt enabled auto-merge (squash) July 10, 2024 05:50

toschdev approved these changes Jul 10, 2024

View reviewed changes

julienrbrt merged commit 31a9c2c into release/v28.x.y Jul 10, 2024
47 of 48 checks passed

julienrbrt deleted the mergify/bp/release/v28.x.y/pr-4133 branch July 10, 2024 09:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improve buf rate limit (backport #4133) #4206

feat: improve buf rate limit (backport #4133) #4206

mergify bot commented Jun 21, 2024

mergify bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024 •

edited

Loading

codecov bot commented Jul 4, 2024 •

edited

Loading

julienrbrt left a comment •

edited

Loading

julienrbrt left a comment

Pantani commented Jul 10, 2024

Pantani commented Jul 10, 2024

feat: improve buf rate limit (backport #4133) #4206

feat: improve buf rate limit (backport #4133) #4206

Conversation

mergify bot commented Jun 21, 2024

Description

mergify bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024 • edited Loading

codecov bot commented Jul 4, 2024 • edited Loading

Codecov Report

julienrbrt left a comment • edited Loading

Choose a reason for hiding this comment

julienrbrt left a comment

Choose a reason for hiding this comment

Pantani commented Jul 10, 2024

Pantani commented Jul 10, 2024

github-actions bot commented Jun 21, 2024 •

edited

Loading

codecov bot commented Jul 4, 2024 •

edited

Loading

julienrbrt left a comment •

edited

Loading