Feature/sg 912 duplicates

[shaydeci]

Finally, in PR.

Do we want to add it as a deafult feature in our configs ?
This would result in always asking the user for the train and validation directory (currently, as we cant automatically derrive them).

[Louis-Dupont]

I left some comments inline.
Can you also run this script https://github.com/Deci-AI/data-gradients/blob/master/scripts/describe_features.py?
This will automatically update the documentation page for the features.

Do we want to add it as a deafult feature in our configs ?
This would result in always asking the user for the train and validation directory (currently, as we cant automatically derive them).

That's a good question... Forcing the user to input the path of train/val would not be a nice experience. So I guess maybe we should just not add it by default. But if we don't add it in the default feature extractors we should somehow indicate the user that this feature is available. Not sure what's the right way

[ofrimasad]

looks great. just minor comments

[ofrimasad]

LGTM. great work