WandB batch metrics logging error

[ivan-chai]

🐛 Bug Report

In wandb all batch metrics are logged as single value per epoch.

Expected behavior

Batch metrics must be logged once per step.

Catalyst version: 21.7

Additional context

The problem is here:

https://github.com/catalyst-team/catalyst/blob/master/catalyst/loggers/wandb.py#L115

Step must be equal to global_sample_step, not global_epoch_step.

[github-actions]

Hi! Thank you for your contribution! Please re-check all issue template checklists - unfilled issues would be closed automatically. And do not forget to join our slack for collaboration.

[Scitator]

maybe, @AyushExel could help 👀

[AyushExel]

• @Scitator I'm out on a vacation. I'll have a look at this on monday. But it sounds like we might not have a good solution for this as W&B only has one global step.

[AyushExel]

@ivan-chai I remember working on this. the reason that the batch metrics are also logged on epoch level is that W&B supports 1 global step and you cannot edit or log at previous steps, so I decided to log everything on epoch level. There are other methods that will end up dropping data if any previous step is encountered.

[ivan-chai]

Thank you for your response! May be it is better to always log sample_step?

[Scitator]

yeah, I think, we could just use global_sample_step as a wandb step for all .log

[AyushExel]

@Scitator @ivan-chai Is the global_sample_step strictly increaseing? If it is, then yes we can use that

[Scitator]

Yes ;)
let's use it

[Scitator]

@AyushExel could you please drop a PR with such a small hotfix?
we are going to release 21.09 version soon ;)

[AyushExel]

@Scitator sounds good. I'll do it within 2 days

[AyushExel]

@Scitator I'm working on it now. I noticed that the logger API now supports artifacts as well. I'll add that too. I just need a bit of clarification on the usage.

    def log_artifact(
        self,
        tag: str,
        artifact: object = None,
        path_to_artifact: str = None,

here, artifact: object is an identifier for the artifact right? Or it can be an in-memory object?
I saw this example usage here

[Scitator]

yup, looks like so

artifact: object = None,  - in-memory object
path_to_artifact: str = None, - on-disk object

here is another example -

catalyst/catalyst/loggers/neptune.py

Line 281 in e1d78b7

if artifact is not None and path_to_artifact is not None:

[Scitator]

btw, @ditwoo why do we use the profiler in such a way? 😂

[AyushExel]

@Scitator in the 2nd example that you linked, it'll raise exception if both artifact and oath_to_artifact is not None. But in the profiler example, both of them are not None so it'll raise an exception. Is this an intended use case?
More specifically, how'd you preferred handling the cases when both of the arguments are set?

[Scitator]

@AyushExel yes, and I think it's correct behavior.
The profiler case looks a bit strange in such a case and I think, it's a little bug 😅
@ditwoo could you please help with the profiler initial logic?

[AyushExel]

Okay, thanks for the clarification. I'll update the logger with the intended artifacts use case.

[AyushExel]

@Scitator I have fixed this and added artifacts support. Can you provide an example training script using catalyst trainer which log artifacts? I want to check all the use cases before submitting the PR. Sorry, I couldn't find something relevant in the quickstart section of docs.

[Scitator]

@AyushExel could you please use https://github.com/neptune-ai/examples/blob/main/integrations-and-supported-tools/catalyst/scripts/Neptune_Catalyst_more_options.py ?

[Scitator]

btw, we are going to release another version tomorrow, so, the fix would truly welcome ;)

[AyushExel]

Just running tests. PR coming soon