[flyte deck] Streaming Decks

[Future-Outlier]

Tracking issue

flyteorg/flyte#5574

Why are the changes needed?

we want to support streaming decks.

What changes were proposed in this pull request?

add a function publish for the Flyte Deck.

How was this patch tested?

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "6b55930d0a77efc3594ebaac056f2c75024e61b5"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=False, container_image=custom_image)
def t_no_deck():
    # Deck.publish()
    print("No Deck")

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(3):
        Deck.publish()
        time.sleep(1)

@task(enable_deck=True, container_image=custom_image)
def t_fail_deck():
    import time

    for i in range(3):
        Deck.publish()
        time.sleep(3)
    time.sleep(10)
    raise ValueError("Failed Deck")

@workflow
def wf():
    t_no_deck()
    t_deck()
    t_fail_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    result = runner.invoke(pyflyte.main,
                           ["run", path, "t_no_deck"])
    print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    print("Remote Execution: ", result.output)

Setup process

single binary.

flyte: flyteorg/flyte#6053
flytekit: #2779
flyteconsole: flyteorg/flyteconsole#890

Screenshots

flytekit: this branch

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-small.mov

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-OLD-FLYTEKIT-small.mov

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

Summary by Bito

Implements comprehensive deck functionality improvements including Deck.publish() method and generates_deck flag handling, with enhanced error messaging. Added task name context to warning messages when enable_deck is False for clearer feedback. Refactors Protocol Buffer BoolValue wrapper from base task to models layer, improving error handling and attribute validation. Changes simplify task definition interface while maintaining proper protobuf serialization.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 3

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.50%. Comparing base (4cee7cd) to head (c0fd23a).
Report is 4 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2779       +/-   ##
===========================================
+ Coverage   77.90%   90.50%   +12.60%     
===========================================
  Files         217      108      -109     
  Lines       22178     4760    -17418     
  Branches     2768        0     -2768     
===========================================
- Hits        17277     4308    -12969     
+ Misses       4115      452     -3663     
+ Partials      786        0      -786     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kumare3]

I would love to see an example of tqdm

[kumare3]

replaces #1704

[Future-Outlier]

I would love to see an example of tqdm

will try it, just let me ship the MSGPACK IDL first.

[EngHabu]

This will make it automatically refresh the page.. probably need that on the empty deck but not sure how to do the realtime behavior once it has content (you don't want to refresh while the user is browsing around or clicking through various tabs)

Suggested change

	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta http-equiv="refresh" content="5" >

[Future-Outlier]

COOL, will try it, thank you

[eapolinario]

This is pretty cool. It unlocks so many use cases!

I left just a couple of comments. We're very close.

[eapolinario]

nice.

[eapolinario]

let's make this error message less charged. How about we say "Call to Deck.publish() will not generate flyte decks as enable_deck=False"?

[eapolinario]

why do we need this?

[Future-Outlier]

because when disable deck, we shouldn't append deck.

[eapolinario]

there's something going on with the serialization tests (e.g.: https://github.com/flyteorg/flytekit/actions/runs/13126446467/job/36627463041?pr=2779). Can you take a look?

[Future-Outlier]

there's something going on with the serialization tests (e.g.: https://github.com/flyteorg/flytekit/actions/runs/13126446467/job/36627463041?pr=2779). Can you take a look?

yes no problem

[flyte-bot]

Code Review Agent Run #1c8466

Actionable Suggestions - 0

Review Details

• Files reviewed - 1 · Commit Range: ea8b6e0..a642c9d

  • flytekit/deck/deck.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[flyte-bot]

Code Review Agent Run #4926be

Actionable Suggestions - 0

Review Details

• Files reviewed - 2 · Commit Range: a642c9d..e9aef35

  • flytekit/tools/translator.py
  • tests/flytekit/unit/deck/test_deck.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[flyte-bot]

Code Review Agent Run #bb9139

Actionable Suggestions - 4

• tests/flytekit/unit/deck/test_deck.py - 2

  • Consider consolidating duplicate context assertions · Line 270-276
  • Consider consolidating repeated assertions · Line 282-301
• flytekit/core/context_manager.py - 1

  • Missing return type hint for builder · Line 153-154
• flytekit/deck/deck.py - 1

  • Consider attribute existence check for params · Line 90-90

Review Details

• Files reviewed - 8 · Commit Range: e9aef35..b649f8f

  • flytekit/core/base_task.py
  • flytekit/core/constants.py
  • flytekit/core/context_manager.py
  • flytekit/core/type_engine.py
  • flytekit/deck/deck.py
  • flytekit/remote/remote.py
  • tests/flytekit/unit/core/test_unions.py
  • tests/flytekit/unit/deck/test_deck.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[flyte-bot]

Code Review Agent Run #c1727c

Actionable Suggestions - 1

• flytekit/core/base_task.py - 1

  • Consider using BoolValue for generates_deck · Line 180-180

Review Details

• Files reviewed - 3 · Commit Range: b649f8f..0565282

  • flytekit/core/base_task.py
  • flytekit/models/task.py
  • tests/flytekit/unit/test_translator.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[flyte-bot]

Code Review Agent Run #a1d78e

Actionable Suggestions - 0

Review Details

• Files reviewed - 1 · Commit Range: 0565282..c0fd23a

  • flytekit/deck/deck.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by