Showing progress with tqdm not working for pl.*.each

[jagin]

Hi,

I would like to use pl.thread.each as my stage is returning nothing but I would like to see some progress:

import pypeln as pl
from tqdm import tqdm

total = 3000


def f(x):
    i = 0
    for _ in range(5000):
        i += 1


stage = range(total)
stage = pl.thread.each(f, stage, workers=8, maxsize=2000)
stage = tqdm(stage, total=total, desc="pipeline")

pl.process.run(stage)

but progress is not working, showing:
pipeline: 0%| | 0/3000 [00:00<?, ?it/s]

With pl.thread.map it is working fine.

[cgarciae]

Hey @jagin !

pl.*.each is a small optimization over pl.*.map which produces an empty iterable, that is, workers don't yield any objects to the output queue. This is why tqdm is not showing any progress.

To track progress you can either use pl.thread.map and use an f without a return (returns None) so that tqdm receives something (this is what I normally do). If you do want to use pl.thread.each, you can create the tqdm object outside f without an iterable (bar = tqdm(total=total, desc="pipeline") and manually update it inside f via bar.update(). This last solution will not work with the pl.process module as it assumes shared memory.

[cgarciae]

Closing this issue for now.