[Go SDK] Timers with new datalayer #26101

riteshghorse · 2023-04-04T13:32:16Z

Timers end to end.

This includes Invoker refactoring and State and Timer example (timer_wordcap) from @lostluck's PR: #25683

Set a timer in ProcessElement
Invoke OnTimerFn
Set timer in OnTimerFn
Infer coder from timer key rather than assuming string coder always
Fn Validation for OnTimer method
Integration test

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

lostluck · 2023-04-04T16:37:47Z

sdks/go/pkg/beam/core/graph/fn.go

+type PipelineTimer interface {
+	TimerFamily() string
+	TimerDomain() timers.TimeDomainEnum
+}


What do you think about switching this to just a "Timers() map[string]timer.Domain" method, to enable the composites I proposed? #25894

We wouldn't need to make the State change in this PR (way way out of scope), but now is ideal for enabling this for timers.

lostluck

Minor suggestions at this stage, largely in the error checking messaging. Very excited to see this moving forward!

sdks/go/pkg/beam/core/graph/fn.go

sdks/go/pkg/beam/core/runtime/exec/datasource.go

sdks/go/pkg/beam/core/runtime/exec/timers.go

hnnsgstfssn

Really exciting to see this take shape and looking forward to working timers in the Go SDK.

I've tried to take a stab at testing this on Dataflow with a few of the use-cases I'm imagining, but have initially run into a few snags that I'm confused about. It may very well be that I'm just misunderstanding how it is supposed to work so bear with me while I put together some more detailed examples to reproduce.

hnnsgstfssn · 2023-04-20T12:22:45Z

sdks/go/examples/timer_wordcap/wordcap.go

+	}
+}
+
+func (s *Stateful) OnTimer(ctx context.Context, ts beam.EventTime, tp timers.Provider, key, timerKey, timerTag string) {


This example only includes the key. If I want to access the value in OnTimer, will I have to pass it through the state API? I see that the design says a value should be mapped, but this says the interaction is through state API. I have not been able to add an input value, getting some variant of

panic: reflect: Call using zero Value argument goroutine 1472 [running]: runtime/debug.Stack() /usr/lib/go/src/runtime/debug/stack.go:24 +0x65 github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic.func1() /home/rru/go/pkg/mod/github.com/riteshghorse/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/util.go:58 +0xa5 panic({0x1d9c940, 0xc000c39260}) /usr/lib/go/src/runtime/panic.go:884 +0x213 reflect.Value.call({0x2016b60?, 0xc000a4d480?, 0x44f6d2?}, {0x28561b5, 0x4}, {0xc000a89970, 0x7, 0x5ddc11?}) /usr/lib/go/src/reflect/value.go:437 +0x1aee reflect.Value.Call({0x2016b60?, 0xc000a4d480?, 0x40e427?}, {0xc000a89970?, 0x24e75a0?, 0xc0006e6801?}) /usr/lib/go/src/reflect/value.go:370 +0xbc github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx.(*reflectFunc).Call(0xc000a86ed0, {0xc000931340?, 0xc0006e6988?, 0xa6660f?}) /home/rru/go/pkg/mod/github.com/riteshghorse/beam/sdks/[email protected]/go/pkg/beam/core/util/reflectx/call.go:87 +0x59 github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).initCall.func46({0x0, 0x0, 0x0, 0x0, 0x0}, {0x3ea9da0, 0x1, 0x1}, 0xf?) /home/rru/go/pkg/mod/github.com/riteshghorse/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/fn_arity.go:307 +0x94 github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).invokeWithOpts(0xc000b6a780, {0x2b630a8?, 0xc000a4ccc0}, {0x0, 0x0, 0x0, 0x0, 0x0}, {0x3ea9da0, 0x1, ...}, ...)

Similarly I have not been able to add any emitter, getting similar errors as per above.

Another issue I had was that it failed to validate when the beam.EventTime was placed after the timers.Provider. It says then that it needs to be before the main input. Moving it into the same position as here seems to work.

Finally I was unable to add timers to a GBK DoFn, having it complain about needing to use KV with timers, but I need to investigate further on this one. Maybe it is not supposed to be possible to use timers in such cases?

Thanks for your interest!

The design doc is a little out of date, and was authored with a less than complete understanding of timers. The Beam Model doc is correct and any values would need to be carried over via the state API.

The FnAPI simply doesn't have a provision for the additional data.

Timers and State are implicitly per key, but timers can have an additional "tag" associated with them for an additional uniqueness factor.

We definitely do need to have emitters working (which would always match the ProcessElement emitters exactly), and it's probably simply just not implemented in this WIP PR. Adding new "lifecycle" methods isn't simple.

I believe that GBKs should be able to have a timer callback and associated state it's largely just a validation thing on the SDK that needs updating.

Thanks for testing it out. I got diverted to other part of the work. I've added support for emitters now. Feel free to test your pipelines

riteshghorse · 2023-04-25T19:32:17Z

@lostluck could you also take a look now?

Regarding the composites, are you proposing a single interface method that would return map[string]TimeDomain? And this method would return just a single KV pair for the built-in timer types (event-time, processing-time) but would be customizable by users in case they want to batch them.

github-actions · 2023-04-25T19:34:52Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label go.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

lostluck · 2023-04-25T20:42:45Z

@lostluck could you also take a look now?

Ack will do! Thanks!

Regarding the composites, are you proposing a single interface method that would return map[string]TimeDomain? And this method would return just a single KV pair for the built-in timer types (event-time, processing-time) but would be customizable by users in case they want to batch them.

That's right. A single method for the detection interface, and optionally a few reflection based helper functions for the composition bit. Basically it unifies the two current methods by packaging them into a map by default, but are returning the same information: The timer-familyID and its associated domain.

lostluck · 2023-04-25T20:42:59Z

stop reviewer notifications

github-actions · 2023-04-25T20:44:06Z

Stopping reviewer notifications for this pull request: requested by reviewer

lostluck · 2023-04-25T20:48:10Z

I guess to be fully concrete: The value of the composites won't be realized until we do it to state too (which will mean eventually deprecating the exist litany of methods, but the principle is the same as timers.)

The state change, and the composition helpers don't need to happen in this PR.

lostluck

I'm already very happy with this. Largely obvious nits, requests for testing, and example cleanup.

sdks/go/examples/timer_wordcap/wordcap.go

sdks/go/pkg/beam/core/runtime/exec/datasource.go

sdks/go/pkg/beam/core/runtime/exec/pardo.go

sdks/go/pkg/beam/core/timers/timers.go

sdks/go/examples/timer_wordcap/wordcap.go

sdks/go/pkg/beam/core/timers/timers.go

riteshghorse · 2023-04-26T17:45:26Z

Thanks for the detailed review @lostluck . I've addressed the comments, PTAL

lostluck

OK, One note about duplicate IDs error message, and then a trail of investigation about the coder used for the timers. We can simplify things by just making what we already know to be true (that it's only the key coder), explicit, and moving several things to be at plan construction time, instead of at invocation time.

Then I think we're done? Exciting!

sdks/go/pkg/beam/core/graph/fn.go

sdks/go/pkg/beam/pardo.go

sdks/go/pkg/beam/core/runtime/exec/translate.go

sdks/go/pkg/beam/core/runtime/exec/coder.go

sdks/go/pkg/beam/core/runtime/exec/timers.go

riteshghorse · 2023-04-27T21:06:31Z

Done the coder related refactoring, PTAL

lostluck

Thanks for all the effort here! I think this is good enough to merge in, which will make it easier to do any last bits around testing and clean up.

You worked for it, I'll let you do the honours.

riteshghorse · 2023-04-28T13:14:48Z

Thanks for the reviews and the very important data layer rewrite!

* added timer package * add timer changes and merged with rebo's pr * timer fired in stateful * error setting new timer in ontimer * looping timers work * send fv instead of bytes * changes to coder/pardo * works for all cases, only cleanup left * remove comments and validate onTimer * generic coder for user key * fixes coder end to end * remove logs * add unit test and refactor * add docs * new example * fix static lint * support emitters * allow input col of CoGBK as well * unit tests, periodic impulse, minor refactor * update PipelineTimer interface, minor refactor, doc comment for example * add warn message * single edge timer coder, rm kv coder check, cache encoder,decoder

* Add channel split separations. * Use channel leases for parallel processing. * move data wait to chan + atomic * Progress and split on execute goroutine. * Convert metrics tests to short ids. * Add Provision handler w/ capabilities. * Add artifact validation to run_rc_validation (#26407) * Add artifact validation to run_rc_validation * config file for artifacts * use java bom contents for validation * add pip_pre = True * Update tox.ini * Remove out of date experimental warning on SDFs (#26450) * [ToBF] Refinement 26.04.23 (#26428) * content tree rebuilds on sdk change * expandable parent node widget * comment fix * comment fix --------- Co-authored-by: darkhan.nausharipov <[email protected]> * Allow implicit flattening for yaml inputs. (#26423) * Add field annotations for high-priority Syndeo schema transforms (#26384) add changes * [Go SDK] Timers with new datalayer (#26101) * added timer package * add timer changes and merged with rebo's pr * timer fired in stateful * error setting new timer in ontimer * looping timers work * send fv instead of bytes * changes to coder/pardo * works for all cases, only cleanup left * remove comments and validate onTimer * generic coder for user key * fixes coder end to end * remove logs * add unit test and refactor * add docs * new example * fix static lint * support emitters * allow input col of CoGBK as well * unit tests, periodic impulse, minor refactor * update PipelineTimer interface, minor refactor, doc comment for example * add warn message * single edge timer coder, rm kv coder check, cache encoder,decoder * Update chromedriver-binary requirement in /sdks/python Updates the requirements on [chromedriver-binary](https://github.com/danielkaiser/python-chromedriver-binary) to permit the latest version. - [Release notes](https://github.com/danielkaiser/python-chromedriver-binary/releases) - [Commits](danielkaiser/python-chromedriver-binary@v100.0.4896.20.0...v113.0.5672.24.0) --- updated-dependencies: - dependency-name: chromedriver-binary dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> * Automation: Tour of Beam infrastructure deployment (#25793) * Added Terraform scripts for TOB infra * ToB Frontend related updates * Update settings.gradle.kts * Deleted redundant file and minor README change * Addressing comments in the PR * Added newline at the end of variables.tf file * Update README.md * Updates related to Tour of Beam infrastructure * Update locals.tf * Output.tf updates * Update output.tf * Updates * Update main.tf * Updates to cloudfunctions_bucket variable * service_account_id changes * Update main.tf * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Bulk update of terraform scripts * Update README.md * Update README.md * Datastore_namespace updates * Update README.md * Update README.md * Update README.md * Update README.md * Update main.tf * Update README.md * Update README.md * Update README.md * Some minor TF updates * Update README.md * Modify batch IT to use count instead of hash (#26327) * Modify batch IT to use count instead of hash * remove unused varaiable * run spotless, update pipeline state checking * Update timeout to 45 minutes * revert timeout, add additional counter to try and pinpoint missing records * add a log to notify ranges used when workers restart * change counts from metrics to combiners * add a window to streaming test * move the passert to the correct place * Remove extra counter, apply spotless * add additional metric to KafkaWriter * Remove debugging metrics * verify pipeline is not failed * remove extra newline * Revert "Modify batch IT to use count instead of hash (#26327)" (#26466) This reverts commit 9903b2f. * Bump Java Dataflow container images (#26459) * keep retrying mass_comment until it has started all jobs (#26457) * keep retrying mass_comment until it has started all jobs * fix lookups * Add driverJars parameter to JdbcIO. (#25824) This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO. * [Roll Fwd PR] Rename _namespace to _get_display_data_namespace"" (#26470) * Update parquetio and textio to work with -beam_strict (#26469) * use wheel sdk location for PostCommit_Py_Examples (#26473) * Move back the timeout of Python PostCommit to 4h * Minor fix on Python PostCommit description strings * Add recent postcommits to jenkins README * More user-friendly providers. * Add yaml preprocessing phases. * Add flexible windowing syntax to yaml. * Implement flatten in terms of preprocessor phase. This composes better with windowing. * Reword SQL note. * Make linter happy. * Survive errors in size estimation in MongoDbIO * Fix jdbc xlang schema type mismatch (#26480) * Fix jdbc xlang schema type mismatch * Also fix fetch_size type mismatch * Add new fields in the end * [Python] Add saved_weights example to tf notebook (#26472) * add saved_weights example to tf notebook * add description * updated text blocks * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <[email protected]> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <[email protected]> --------- Co-authored-by: Rebecca Szper <[email protected]> * Revert "Revert "Modify batch IT to use count instead of hash (#26327)" (#26466)" (#26467) This reverts commit 5c67668. * Bump github.com/tetratelabs/wazero from 1.0.3 to 1.1.0 in /sdks (#26486) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.0.3 to 1.1.0. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](tetratelabs/wazero@v1.0.3...v1.1.0) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * tour of beam integration tests (#25925) * Integration test to load the default example of the default SDK and change the example (#24730) (#24729) * Fix formatting and README (#24730) * Support collection v1.17.0 (#24730) * LoadingIndicator on chaning examples, remove duplicating licenses (#24730) * Add a missing license header (#24730) * Integration test for changing SDK and running code (#24779) (#382) * Integration test for changing SDK and running code (#24779) * Rename an integration test (#24779) * Use enum to switch SDK in integration test (#24779) * Find SDK in a dropdown by key (#24779) * Add a TODO (#24779) * Fix exports (#24779) * Issue24779 integration changing sdk from 24370 (#387) * Integration test for changing SDK and running code (#24779) * Rename an integration test (#24779) * Use enum to switch SDK in integration test (#24779) * Find SDK in a dropdown by key (#24779) * Add a TODO (#24779) * Fix exports (#24779) * Integration tests miscellaneous UI (#383) * miscellaneous ui integration tests * reverted pubspec.lock * gradle tasks ordered alhpabetically * integration tests refactoring * clean code * integration tests miscellaneous ui fix pr * rename method * added layout adaptivity * A minor cleanup (#24779) Co-authored-by: Dmitry Repin <[email protected]> * integration tests run and editing * example selector test * minor fixes * rat * fix pr * minor * minor * rat * integration test finder written * integration test minor fixes * minor fixes * removed comment * minor fixes * playground integration tests minor fixes * integration test pumpAnSettleNoException * integration test shortcut refactor * integration test another changing shortcuts running * upgrade to flutter 3.7.1 * workaround comment * playground frontend updated major versions * issues 25329 25331 25336 * 25329 extract connectivity extension to separate file * Upgrade Flutter to 3.7.3 in integration tests (#24730) * Fix integration test (#24730) * fix cors issue and added mouse scroll to tags * Upgrade Flutter in Dockerfile (#24720) * sorting moved to model * sorting moved to model * sorting moved to model * bugs fix * issue 25278 * fix pr * quites fix in en.yaml * Fix not loading default example (#25528) * fix compile error * Refactor output tabs, test embedded playground (#25136) (#439) * Refactor output tabs, test embedded playground (#25136) * Clean up (#25136) * Change example paths to IDs in integration tests * issue25640 tob ci * fix tob ci * rename ci process * test add new line to main * test add new line to main * commented unit test run * issue25640 changed server path * issue25640 tests on welcome page * deleted config.g.dart * issue25640 pr fixes * Update .github/workflows/tour_of_beam_frontend_test.yml Co-authored-by: alexeyinkin <[email protected]> * Update learning/tour-of-beam/frontend/integration_test/welcome_page_test.dart Co-authored-by: alexeyinkin <[email protected]> * Improve tests (#25640) * issue25640 tour page tests * pr fix * removed import * pr fix * fix test * 25640 fixed pubspec.lock * issue25640 fix readme * updated readme * issue25640 fixed after master merge * issue25483 ToB pipeline options * removed unnecesary variable * pr fix * Update learning/tour-of-beam/frontend/assets/translations/en.yaml Co-authored-by: alexeyinkin <[email protected]> * playground hides when snippet does not exists * pipeline options extracted to playground components * issue25483 pipeline options * added errors handling, fix pr * refactoring * Revert "refactoring" This reverts commit 1540961. * removed unnecessary constants * playground controller in tour notifier becomes nullable * playground controller returned to non nullable in tour notifier * playground controller actions * removed unnecessary code * tob scaffold wrapped with animated builder * minor fixes * partially fixed tests * Upgrade flutter_code_editor to v0.2.19 (#25640) * Replace output SelectableText with a CodeField instance (#25640) * Trigger ToB integration tests (#25640) * Clean up (#25640) * Enable manual workflow runs for Playground and ToB integration tests (#25640) --------- Co-authored-by: Alexey Inkin <[email protected]> Co-authored-by: alexeyinkin <[email protected]> * Eliminate nullness errors from MongoDbIO * Move provisioned options outside of harness.Main (#26476) Co-authored-by: lostluck <[email protected]> * Add SDF adjacent element test * Consolidate residual processing * fix race condition on split boundaries * deflake fixed window combine tests * Add comments to job functions. * Make int64check per window. * Update sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go Co-authored-by: Ritesh Ghorse <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: lostluck <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Anand Inguva <[email protected]> Co-authored-by: Anand Inguva <[email protected]> Co-authored-by: Darkhan Nausharipov <[email protected]> Co-authored-by: darkhan.nausharipov <[email protected]> Co-authored-by: Robert Bradshaw <[email protected]> Co-authored-by: Andrei Gurau <[email protected]> Co-authored-by: Ritesh Ghorse <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ruslan-ikhsan <[email protected]> Co-authored-by: johnjcasey <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Pranav Bhandari <[email protected]> Co-authored-by: Svetak Sundhar <[email protected]> Co-authored-by: Jeremy Edwards <[email protected]> Co-authored-by: Kenneth Knowles <[email protected]> Co-authored-by: Rebecca Szper <[email protected]> Co-authored-by: Dmitry Repin <[email protected]> Co-authored-by: Alexey Inkin <[email protected]> Co-authored-by: alexeyinkin <[email protected]>

* Add channel split separations. * Use channel leases for parallel processing. * move data wait to chan + atomic * Progress and split on execute goroutine. * Convert metrics tests to short ids. * Add Provision handler w/ capabilities. * Add artifact validation to run_rc_validation (apache#26407) * Add artifact validation to run_rc_validation * config file for artifacts * use java bom contents for validation * add pip_pre = True * Update tox.ini * Remove out of date experimental warning on SDFs (apache#26450) * [ToBF] Refinement 26.04.23 (apache#26428) * content tree rebuilds on sdk change * expandable parent node widget * comment fix * comment fix --------- Co-authored-by: darkhan.nausharipov <[email protected]> * Allow implicit flattening for yaml inputs. (apache#26423) * Add field annotations for high-priority Syndeo schema transforms (apache#26384) add changes * [Go SDK] Timers with new datalayer (apache#26101) * added timer package * add timer changes and merged with rebo's pr * timer fired in stateful * error setting new timer in ontimer * looping timers work * send fv instead of bytes * changes to coder/pardo * works for all cases, only cleanup left * remove comments and validate onTimer * generic coder for user key * fixes coder end to end * remove logs * add unit test and refactor * add docs * new example * fix static lint * support emitters * allow input col of CoGBK as well * unit tests, periodic impulse, minor refactor * update PipelineTimer interface, minor refactor, doc comment for example * add warn message * single edge timer coder, rm kv coder check, cache encoder,decoder * Update chromedriver-binary requirement in /sdks/python Updates the requirements on [chromedriver-binary](https://github.com/danielkaiser/python-chromedriver-binary) to permit the latest version. - [Release notes](https://github.com/danielkaiser/python-chromedriver-binary/releases) - [Commits](danielkaiser/python-chromedriver-binary@v100.0.4896.20.0...v113.0.5672.24.0) --- updated-dependencies: - dependency-name: chromedriver-binary dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> * Automation: Tour of Beam infrastructure deployment (apache#25793) * Added Terraform scripts for TOB infra * ToB Frontend related updates * Update settings.gradle.kts * Deleted redundant file and minor README change * Addressing comments in the PR * Added newline at the end of variables.tf file * Update README.md * Updates related to Tour of Beam infrastructure * Update locals.tf * Output.tf updates * Update output.tf * Updates * Update main.tf * Updates to cloudfunctions_bucket variable * service_account_id changes * Update main.tf * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Bulk update of terraform scripts * Update README.md * Update README.md * Datastore_namespace updates * Update README.md * Update README.md * Update README.md * Update README.md * Update main.tf * Update README.md * Update README.md * Update README.md * Some minor TF updates * Update README.md * Modify batch IT to use count instead of hash (apache#26327) * Modify batch IT to use count instead of hash * remove unused varaiable * run spotless, update pipeline state checking * Update timeout to 45 minutes * revert timeout, add additional counter to try and pinpoint missing records * add a log to notify ranges used when workers restart * change counts from metrics to combiners * add a window to streaming test * move the passert to the correct place * Remove extra counter, apply spotless * add additional metric to KafkaWriter * Remove debugging metrics * verify pipeline is not failed * remove extra newline * Revert "Modify batch IT to use count instead of hash (apache#26327)" (apache#26466) This reverts commit 9903b2f. * Bump Java Dataflow container images (apache#26459) * keep retrying mass_comment until it has started all jobs (apache#26457) * keep retrying mass_comment until it has started all jobs * fix lookups * Add driverJars parameter to JdbcIO. (apache#25824) This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO. * [Roll Fwd PR] Rename _namespace to _get_display_data_namespace"" (apache#26470) * Update parquetio and textio to work with -beam_strict (apache#26469) * use wheel sdk location for PostCommit_Py_Examples (apache#26473) * Move back the timeout of Python PostCommit to 4h * Minor fix on Python PostCommit description strings * Add recent postcommits to jenkins README * More user-friendly providers. * Add yaml preprocessing phases. * Add flexible windowing syntax to yaml. * Implement flatten in terms of preprocessor phase. This composes better with windowing. * Reword SQL note. * Make linter happy. * Survive errors in size estimation in MongoDbIO * Fix jdbc xlang schema type mismatch (apache#26480) * Fix jdbc xlang schema type mismatch * Also fix fetch_size type mismatch * Add new fields in the end * [Python] Add saved_weights example to tf notebook (apache#26472) * add saved_weights example to tf notebook * add description * updated text blocks * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <[email protected]> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <[email protected]> --------- Co-authored-by: Rebecca Szper <[email protected]> * Revert "Revert "Modify batch IT to use count instead of hash (apache#26327)" (apache#26466)" (apache#26467) This reverts commit 5c67668. * Bump github.com/tetratelabs/wazero from 1.0.3 to 1.1.0 in /sdks (apache#26486) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.0.3 to 1.1.0. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](tetratelabs/wazero@v1.0.3...v1.1.0) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * tour of beam integration tests (apache#25925) * Integration test to load the default example of the default SDK and change the example (apache#24730) (apache#24729) * Fix formatting and README (apache#24730) * Support collection v1.17.0 (apache#24730) * LoadingIndicator on chaning examples, remove duplicating licenses (apache#24730) * Add a missing license header (apache#24730) * Integration test for changing SDK and running code (apache#24779) (apache#382) * Integration test for changing SDK and running code (apache#24779) * Rename an integration test (apache#24779) * Use enum to switch SDK in integration test (apache#24779) * Find SDK in a dropdown by key (apache#24779) * Add a TODO (apache#24779) * Fix exports (apache#24779) * Issue24779 integration changing sdk from 24370 (apache#387) * Integration test for changing SDK and running code (apache#24779) * Rename an integration test (apache#24779) * Use enum to switch SDK in integration test (apache#24779) * Find SDK in a dropdown by key (apache#24779) * Add a TODO (apache#24779) * Fix exports (apache#24779) * Integration tests miscellaneous UI (apache#383) * miscellaneous ui integration tests * reverted pubspec.lock * gradle tasks ordered alhpabetically * integration tests refactoring * clean code * integration tests miscellaneous ui fix pr * rename method * added layout adaptivity * A minor cleanup (apache#24779) Co-authored-by: Dmitry Repin <[email protected]> * integration tests run and editing * example selector test * minor fixes * rat * fix pr * minor * minor * rat * integration test finder written * integration test minor fixes * minor fixes * removed comment * minor fixes * playground integration tests minor fixes * integration test pumpAnSettleNoException * integration test shortcut refactor * integration test another changing shortcuts running * upgrade to flutter 3.7.1 * workaround comment * playground frontend updated major versions * issues 25329 25331 25336 * 25329 extract connectivity extension to separate file * Upgrade Flutter to 3.7.3 in integration tests (apache#24730) * Fix integration test (apache#24730) * fix cors issue and added mouse scroll to tags * Upgrade Flutter in Dockerfile (apache#24720) * sorting moved to model * sorting moved to model * sorting moved to model * bugs fix * issue 25278 * fix pr * quites fix in en.yaml * Fix not loading default example (apache#25528) * fix compile error * Refactor output tabs, test embedded playground (apache#25136) (apache#439) * Refactor output tabs, test embedded playground (apache#25136) * Clean up (apache#25136) * Change example paths to IDs in integration tests * issue25640 tob ci * fix tob ci * rename ci process * test add new line to main * test add new line to main * commented unit test run * issue25640 changed server path * issue25640 tests on welcome page * deleted config.g.dart * issue25640 pr fixes * Update .github/workflows/tour_of_beam_frontend_test.yml Co-authored-by: alexeyinkin <[email protected]> * Update learning/tour-of-beam/frontend/integration_test/welcome_page_test.dart Co-authored-by: alexeyinkin <[email protected]> * Improve tests (apache#25640) * issue25640 tour page tests * pr fix * removed import * pr fix * fix test * 25640 fixed pubspec.lock * issue25640 fix readme * updated readme * issue25640 fixed after master merge * issue25483 ToB pipeline options * removed unnecesary variable * pr fix * Update learning/tour-of-beam/frontend/assets/translations/en.yaml Co-authored-by: alexeyinkin <[email protected]> * playground hides when snippet does not exists * pipeline options extracted to playground components * issue25483 pipeline options * added errors handling, fix pr * refactoring * Revert "refactoring" This reverts commit 1540961. * removed unnecessary constants * playground controller in tour notifier becomes nullable * playground controller returned to non nullable in tour notifier * playground controller actions * removed unnecessary code * tob scaffold wrapped with animated builder * minor fixes * partially fixed tests * Upgrade flutter_code_editor to v0.2.19 (apache#25640) * Replace output SelectableText with a CodeField instance (apache#25640) * Trigger ToB integration tests (apache#25640) * Clean up (apache#25640) * Enable manual workflow runs for Playground and ToB integration tests (apache#25640) --------- Co-authored-by: Alexey Inkin <[email protected]> Co-authored-by: alexeyinkin <[email protected]> * Eliminate nullness errors from MongoDbIO * Move provisioned options outside of harness.Main (apache#26476) Co-authored-by: lostluck <[email protected]> * Add SDF adjacent element test * Consolidate residual processing * fix race condition on split boundaries * deflake fixed window combine tests * Add comments to job functions. * Make int64check per window. * Update sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go Co-authored-by: Ritesh Ghorse <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: lostluck <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Anand Inguva <[email protected]> Co-authored-by: Anand Inguva <[email protected]> Co-authored-by: Darkhan Nausharipov <[email protected]> Co-authored-by: darkhan.nausharipov <[email protected]> Co-authored-by: Robert Bradshaw <[email protected]> Co-authored-by: Andrei Gurau <[email protected]> Co-authored-by: Ritesh Ghorse <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ruslan-ikhsan <[email protected]> Co-authored-by: johnjcasey <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Pranav Bhandari <[email protected]> Co-authored-by: Svetak Sundhar <[email protected]> Co-authored-by: Jeremy Edwards <[email protected]> Co-authored-by: Kenneth Knowles <[email protected]> Co-authored-by: Rebecca Szper <[email protected]> Co-authored-by: Dmitry Repin <[email protected]> Co-authored-by: Alexey Inkin <[email protected]> Co-authored-by: alexeyinkin <[email protected]>

riteshghorse added 4 commits March 29, 2023 11:15

added timer package

6cdbcd0

add timer changes and merged with rebo's pr

f0e9ba6

timer fired in stateful

e5ec39b

error setting new timer in ontimer

15386c6

github-actions bot added examples go labels Apr 4, 2023

lostluck reviewed Apr 4, 2023

View reviewed changes

riteshghorse added 12 commits April 5, 2023 10:47

looping timers work

4fe6307

send fv instead of bytes

49ead2b

changes to coder/pardo

efe50cf

works for all cases, only cleanup left

77b1438

remove comments and validate onTimer

3e0e4de

merge master

49407e0

generic coder for user key

4191839

fixes coder end to end

dd0c4e6

remove logs

95d641f

add unit test and refactor

6ce7814

add docs

9762682

new example

5b012f5

riteshghorse marked this pull request as ready for review April 20, 2023 04:58

fix static lint

64d4759

hnnsgstfssn reviewed Apr 20, 2023

View reviewed changes

riteshghorse added 2 commits April 25, 2023 15:15

support emitters

afff87e

allow input col of CoGBK as well

601dd58

riteshghorse changed the title ~~[WIP] Timers with new datalayer~~ [Go SDK] Timers with new datalayer Apr 25, 2023

github-actions bot added the Next Action: Reviewers label Apr 25, 2023

lostluck requested changes Apr 25, 2023

View reviewed changes

riteshghorse added 3 commits April 26, 2023 12:02

unit tests, periodic impulse, minor refactor

6ea3969

update PipelineTimer interface, minor refactor, doc comment for example

06bad75

add warn message

1c9e57b

lostluck reviewed Apr 26, 2023

View reviewed changes

single edge timer coder, rm kv coder check, cache encoder,decoder

161f5a9

lostluck approved these changes Apr 28, 2023

View reviewed changes

riteshghorse merged commit 1bb2a69 into apache:master Apr 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Go SDK] Timers with new datalayer #26101

[Go SDK] Timers with new datalayer #26101

riteshghorse commented Apr 4, 2023 •

edited by lostluck

Loading

lostluck Apr 4, 2023

lostluck left a comment

hnnsgstfssn left a comment

hnnsgstfssn Apr 20, 2023 •

edited

Loading

lostluck Apr 21, 2023

riteshghorse Apr 25, 2023

riteshghorse commented Apr 25, 2023

github-actions bot commented Apr 25, 2023

lostluck commented Apr 25, 2023

lostluck commented Apr 25, 2023

github-actions bot commented Apr 25, 2023

lostluck commented Apr 25, 2023

lostluck left a comment

riteshghorse commented Apr 26, 2023

lostluck left a comment

riteshghorse commented Apr 27, 2023

lostluck left a comment

riteshghorse commented Apr 28, 2023

[Go SDK] Timers with new datalayer #26101

[Go SDK] Timers with new datalayer #26101

Conversation

riteshghorse commented Apr 4, 2023 • edited by lostluck Loading

GitHub Actions Tests Status (on master branch)

lostluck Apr 4, 2023

Choose a reason for hiding this comment

lostluck left a comment

Choose a reason for hiding this comment

hnnsgstfssn left a comment

Choose a reason for hiding this comment

hnnsgstfssn Apr 20, 2023 • edited Loading

Choose a reason for hiding this comment

lostluck Apr 21, 2023

Choose a reason for hiding this comment

riteshghorse Apr 25, 2023

Choose a reason for hiding this comment

riteshghorse commented Apr 25, 2023

github-actions bot commented Apr 25, 2023

lostluck commented Apr 25, 2023

lostluck commented Apr 25, 2023

github-actions bot commented Apr 25, 2023

lostluck commented Apr 25, 2023

lostluck left a comment

Choose a reason for hiding this comment

riteshghorse commented Apr 26, 2023

lostluck left a comment

Choose a reason for hiding this comment

riteshghorse commented Apr 27, 2023

lostluck left a comment

Choose a reason for hiding this comment

riteshghorse commented Apr 28, 2023

riteshghorse commented Apr 4, 2023 •

edited by lostluck

Loading

hnnsgstfssn Apr 20, 2023 •

edited

Loading