[Bug]: Videos duplicate themselves when importing playlists

[Xindaris]

Guidelines

• I have encountered this bug in the latest release of FreeTube.
• I have encountered this bug in the official downloads of FreeTube.
• I have searched the issue tracker for open and closed issues that are similar to the bug report I want to file, without success.
• I have searched the documentation for information that matches the description of the bug I want to file, without success.
• This issue contains only one bug.

Describe the bug

1. Add videos to a playlist. (Using the quick bookmark feature; unsure if that's relevant)
2. Export the playlist.
3. Import the playlist on a different computer.
4. Add videos to the playlist on that other computer, and export again.
5. Import on the first computer.
6. Some videos randomly get two "copies" in the playlist now.

Expected Behavior

Playlists should merge properly when importing, and not add another copy of a video that's already in the playlist.

Issue Labels

inconsistent behavior, usability issue

FreeTube Version

v0.20.0 Beta

Operating System Version

Windows 11

Installation Method

.exe

Primary API used

Local API

Last Known Working FreeTube Version (If Any)

No response

Additional Information

No response

Nightly Build

• I have encountered this bug in the latest nightly build.

[github-actions]

This issue is stale because it has been open 28 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[efb4f5ff-1298-471a-8973-3d47447115dc]

Is this still present in v0.21.0?

[Xindaris]

I've still seen this happening myself, but at least the "delete duplicates" button in the playlist menu works.

[PikachuEXE]

Is the duplicates always created for the same set of videos or complete random?
If always the same set of videos would you mind sharing them? (Video links, ids)

No idea how to reproduce it happens randomly (I tried once myself

[Xindaris]

As far as I can tell, the conditions for me to encounter it are as follows (using two computers):

1. On computer 1, add the video to the playlist and export
2. On computer 2, add the video to the playlist
3. On computer 2, import what computer 1 exported.

This process seems to be necessary but not necessarily sufficient. That is, videos may or may not be duplicated when doing this, but I'm pretty sure I never get duplicates if I, say, don't do the second step above.

[efb4f5ff-1298-471a-8973-3d47447115dc]

As far as I can tell, the conditions for me to encounter it are as follows (using two computers):

1. On computer 1, add the video to the playlist and export

2. On computer 2, add the video to the playlist

3. On computer 2, import what computer 1 exported.

This process seems to be necessary but not necessarily sufficient. That is, videos may or may not be duplicated when doing this, but I'm pretty sure I never get duplicates if I, say, don't do the second step above.

Can confirm this is happening to me using these steps

[github-actions]

This issue is stale because it has been open 28 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue is stale because it has been open 28 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[PikachuEXE]

@efb4f5ff-1298-471a-8973-3d47447115dc
If you can reproduce can you post the exported files from computer 1 & 2 for debug

[efb4f5ff-1298-471a-8973-3d47447115dc]

@PikachuEXE I imported this file on my 2nd vm:

• PC1-freetube-playlists-2024-09-28.zip

VirtualBoxVM_u4tohN9PaY.mp4

This is how the .db looks like after the import:

• PC2-freetube-playlists-2024-09-28.zip

[PikachuEXE]

As far as I can tell, the conditions for me to encounter it are as follows (using two computers):

1. On computer 1, add the video to the playlist and export
2. On computer 2, add the video to the playlist
3. On computer 2, import what computer 1 exported.

This process seems to be necessary but not necessarily sufficient. That is, videos may or may not be duplicated when doing this, but I'm pretty sure I never get duplicates if I, say, don't do the second step above.

Let me see if I understand this correctly
On (1), video (A) added (via UI)
On (2), video (A) added (via UI)
On (3), video (A) imported from computer 1 to computer 2

If this is the case it's intended
FT currently recognizes playlist items based on unique ID (playlistItemId in code) generated per addition
So the playlist item on computer 1 would be like A-11111, on computer 2 would be A-22222
On import FT is putting A-11111 under existing A-22222

Ideally FT should let user decide what user want to do with "duplicate videos"

Let me know if anything missing/incorrect, then we can discuss what to do with it

[efb4f5ff-1298-471a-8973-3d47447115dc]

Let me see if I understand this correctly

I think you hit the nail on its head

If this is the case it's intended

Ok i understand but lets say the user got 50 playlists than its get quite cumbersome to go into every playlist just to hit the de-duplicate button. IMO it shouldnt duplicate on import. We should only allow duplicate videos on import if the exported playlist already contained duplicates.

[PikachuEXE]

#5783

For custom build let @efb4f5ff-1298-471a-8973-3d47447115dc test first before I make one
Or you can try to run dev version locally if you know how to

[PikachuEXE]

@efb4f5ff-1298-471a-8973-3d47447115dc has tested #5783
I would like other people following this issue to test it as well
A custom build is created: https://github.com/PikachuEXE/FreeTube/actions/runs/11189715522 (latest dev + #5783)
Backup data before using any custom build/dev version as usual :)