Consider using libappimageupdate for updating Linux clients

[TheAssassin]

There is an update check in the application that checks for new versions, but the users will have to download the new files themselves. On Linux, however, with AppImage being the main way for distributing the client software, you could use libappimageupdate, a library provided by the AppImage team, to automatically update the AppImage. This is done via the AppImage update mechanisms specified in the AppImage spec, and highly efficient due to the rsync-like delta update.

What has to be done for this is to start embedding so-called update information in the released AppImages. By adding -u 'my update information' to the appimagetool call. This causes appimagetool to generate a file with a .zsync ending. This file needs to be put on a webserver with a fixed URL that will always be available, or, alternatively, you can make use of one of the "automagic" types such as the GitHub releases type etc. This file will point to the new release's location on your servers, from which AppImageUpdate will perform a delta file sync.

For Linux, you could even replace the update check with libappimageupdate, but as it doesn't support meta information (yet), I guess you should combine that with your normal update check to make sure there really is an update available for download.

If you need help, we can assist with this process.

P.S.: We're working on a "standard" Qt UI developers can integrate so they don't have to do it yourself.

[Fuseteam]

what is blocking this feature? is the tooling to generate the appimage part of this repo? it seems issue #2656 would be solved if this gets implemented

[camilasan]

what is blocking this feature? is the tooling to generate the appimage part of this repo? it seems issue #2656 would be solved if this gets implemented

https://github.com/nextcloud/client-building/tree/master/linux

[Fuseteam]

what is blocking this feature? is the tooling to generate the appimage part of this repo? it seems issue #2656 would be solved if this gets implemented

https://github.com/nextcloud/client-building/tree/master/linux

so that's where the proposed changes should be made?

[Fuseteam]

@TheAssassin do i understand correctly do that -u 'my update information have to added here? https://github.com/nextcloud/client-building/blob/b9343adedf89dfec90274beefee4418884dd1b26/linux/build-appimage-daily.sh#L97

[TheAssassin]

I don't provide support for linuxdeployqt.

[Fuseteam]

I don't provide support for linuxdeployqt.

wait so linuxdeployqt does what appimagetool normally does or something?

[antony-jr]

I don't provide support for linuxdeployqt.

wait so linuxdeployqt does what appimagetool normally does or something?

@Fuseteam linuxdeployqt copies all the required shared libraries required to run your target application (the same as macdeployqt or windeployqt which are supported by qt officially) in AppDir format and bundles it as a AppImage optionally. But appimagetool only does the packaging but does not copy qt libraries.

So,

linuxdeployqt generates AppDir specifically for Qt Applications
appimagetool generates AppImage from AppDir

It's also worthy to note that linuxdeployqt uses appimagetool internally to convert the AppDir to AppImage after generating it if you wish to.

[Fuseteam]

@antony-jr oh interesting. so then, if i understand correctly, the change needed to use libappimageupdate would need to need to done in linuxdeployqt?

[antony-jr]

@antony-jr oh interesting. so then, if i understand correctly, the change needed to use libappimageupdate would need to need to done in linuxdeployqt?

Yes.

You just need this argument in the linuxdeployqt program invocation

-updateinformation=<update string>        : Embed update information STRING; if zsyncmake is installed, generate zsync file

This update string should follow this spec https://github.com/AppImage/AppImageSpec/blob/master/draft.md#update-information

A valid update string would look like this,

gh-releases-zsync|antony-jr|AppImageUpdater|development|AppImageUpdater*-x86_64.AppImage.zsync

The delimiter here is the |.

So the first part tells we will fetch the latest version from Github(there are other options, please see the spec).
The second part is the Github user which holds the repo.
The third part is the repo name.
The fourth part is the release tag that you use in the Github releases.
The last part is the name of the zsync file which is generated by linuxdeployqt when provided update information(if zsync is installed using package manager) which is uploaded along with your AppImage.

The release should contain at least the files shown in the below image (YOU DON'T NEED THE TORRENT FILE, IT's JUST MY SPECIAL CASE)

About libappimageupdate

It's a C++ library for updating AppImages with the zsync algorithm. You need to use this if you want the delta updater inside your application. So you have to implement it inside the application. But there are external application to do this job. In this case the user would have to download the updater from https://github.com/AppImage/AppImageUpdate and use it to update your AppImage.

AppImageLauncher which is used a lot has this inbuilt so the user don't have to download or install anything if AppImageLauncher is installed.

EDIT:

@Fuseteam you can take a look on how I deploy my application with the required update information https://github.com/antony-jr/AppImageUpdater/blob/mk2/.github/workflows/deploy.yml

[Fuseteam]

oh wow thanks;for the extensive write up @antony-jr i'll try it in the above script when i find a chance unless someone else picks it up

[dasHorst99]

+1
Is there some progress on this topic? I also would like to see this feature.

[Fuseteam]

Haven't had a chance to look at this yet sadly

[antony-jr]

@Fuseteam you only need to add a single argument and upload the zsync file to releases to enable delta updates.

Here

desktop/admin/linux/build-appimage.sh

Line 97 in d06998a

./squashfs-root/AppRun ${DESKTOP_FILE} -appimage

it needs to be

./squashfs-root/AppRun ${DESKTOP_FILE} -appimage -updateinformation="gh-releases-zsync|nextcloud|desktop|latest|Nextcloud-*-x86_64.AppImage.zsync"

and you need to upload the .zsync file (which will be generated by linuxdeployqt if zsync is installed in the build distro [apt install zsync]) along with your AppImage here

desktop/admin/linux/upload-appimage.sh

Line 72 in d06998a

echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl --header "Content-Type: application/octet-stream" --upload-file $APPIMAGE)

[Fuseteam]

@antony-jr well that seems easy enough, i however don't think i am able to upload files to releases...... If i created a PR thar would be my 'first contribution'

[antony-jr]

@antony-jr well that seems easy enough, i however don't think i am able to upload files to releases...... If i created a PR thar would be my 'first contribution'

Oops, thought you were a member. Sorry.

[Thatoo]

@TheAssassin and @antony-jr could you help us with this PR : #3736 (comment) ?

[Fuseteam]

oooh i see its merged, looks like we'll be getting this in 3.4?

[antony-jr]

oooh i see its merged, looks like we'll be getting this in 3.4?

The original issue is to consider using libappimageupdate for updating linux clients. The merge only embeds the required update information for NextCloud AppImage. So partially yes, you can delta update next cloud appimages from next release using the official AppImage Updater (https://github.com/AppImage/AppImageUpdate). But to do this without any third party application and directly within the next cloud application itself then they would have to use libappimageupdate. But IMO this is better since AppImageLauncher has the AppImage update built into it so it will give menu options to update next cloud automatically.

[Fuseteam]

ah cool thanks for the clarification

[Thatoo]

I trtied to update from 3.4.0 to 3.4.1 with the help of AppImageLauncher but I get this error :

Fetching latest release information from GitHub API
None of the artifacts matched the pattern in the update information. The pattern is most likely invalid, e.g., due to changes in the filenames of the AppImages. Please contact the author of the AppImage and ask them to revise the update information.
Fetching latest release information from GitHub API
None of the artifacts matched the pattern in the update information. The pattern is most likely invalid, e.g., due to changes in the filenames of the AppImages. Please contact the author of the AppImage and ask them to revise the update information.
ZSync URL not available. See previous messages for details.

😞

[antony-jr]

@Thatoo Just as the error message says, the zsync file is missing in the release, the CI/CD should upload the .zsync file along with the AppImage. You can generate the .zsync file of the AppImage with a tool called zsync but linuxdeployqt does this automatically so just need to upload it.

The file Nextcloud-3.4.1-x86_64.AppImage.zsync is missing from the assets.

[Fuseteam]

looks like it is included in the script

[antony-jr]

@Fuseteam True but that's the issue why it's not working. We need to resolve it somehow. Someone who has access to the entire CI/CD pipeline and knowledge on how the release works might be able to fix this soon. I really have no clue how they do this and don't have time for that. 😕

[Fuseteam]

true same here, it must be missed somewhere hmmm

[Thatoo]

maybe @mgallien or @FlexW could help us find out why it isn't working.

[Thatoo]

@antony-jr or @TheAssassin could it be linked to this issue not being solved #2848?
Should we add export VERSION=... just after

desktop/admin/linux/upload-appimage.sh

Line 28 in 3ab698a

export BASENAME=$(basename ${APPIMAGE})

?

[antony-jr]

@antony-jr or @TheAssassin could it be linked to this issue not being solved #2848? Should we add export VERSION=... just after

desktop/admin/linux/upload-appimage.sh

Line 28 in 3ab698a

export BASENAME=$(basename ${APPIMAGE})

?

No it's completely unrelated. The issue as it stands today is pretty simple. There is no .zsync file uploaded to the assets in the releases.

This line

desktop/admin/linux/upload-appimage.sh

Line 74 in 3ec9c00

echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl --header "Content-Type: application/octet-stream" --upload-file $UPDATE)

looks very suspicious.

The upload url changes when you upload a file with different file name. But it seems you never create a new upload url(with the filename changed) for the zsync file and call upload release asset.

[Thatoo]

the line 74 looks really similar to line 73, no?
And I got inspiration from #168 (comment)
So I modified build-appimage.sh and upload-appimage.sh the way I understood your post but I could have misunderstood it.
I hope we can find a solution.
This is my only contribution so far. I'm not a big contributor, just trying to help when I can.

[Fuseteam]

i love how i learn something new every time i get notif here xD
so basically when the appimage is generated a .zsync file should also be generated.
and that file should be uploaded to https://github.com/nextcloud/desktop/releases/latest

[Fuseteam]

taking another look

desktop/admin/linux/upload-appimage.sh

Line 27 in c2f72b5

export UPDATE=$(readlink -f ./Nextcloud*.AppImage.zsync)

looking at line 27 it sounds like it's not seeing/finding ./Nextcloud*.AppImage.zsync if it's generated at all

[Fuseteam]

wait no i'm stupid; anthony already pointed it out

desktop/admin/linux/upload-appimage.sh

Lines 26 to 28 in c2f72b5

	export APPIMAGE=$(readlink -f ./Nextcloud*.AppImage)
	export UPDATE=$(readlink -f ./Nextcloud*.AppImage.zsync)
	export BASENAME=$(basename ${APPIMAGE})

desktop/admin/linux/upload-appimage.sh

Line 103 in c2f72b5

uploadUrl=$(echo "${uploadUrl/'{?name,label}'/?name=$BASENAME}")

desktop/admin/linux/upload-appimage.sh

Line 123 in c2f72b5

json=$(upload_release_asset "$uploadUrl")

uploadUrl is only valid for $BASENAME which is based on $APPIMAGE not on $UPDATE

desktop/admin/linux/upload-appimage.sh

Lines 70 to 75 in c2f72b5

	upload_release_asset()
	{
	uploadUrl=$1
	echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl --header "Content-Type: application/octet-stream" --upload-file $APPIMAGE)
	echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl --header "Content-Type: application/octet-stream" --upload-file $UPDATE)
	}

so basically upload_release_asset should take two uploadurls as argument, one for $APPIMAGE and one for $UPDATE
maybe something like this:

 export BASENAME_APPIMAGE=$(basename ${APPIMAGE})
 export BASENAME_UPDATE=$(basename ${UPDATE})

 uploadUrlAppImage=$(echo "${uploadUrl/'{?name,label}'/?name=$BASENAME_APPIMAGE}")
 uploadUrlUpdate=$(echo "${uploadUrl/'{?name,label}'/?name=$BASENAME_UPDATE}")

 upload_release_asset() 
 { 
     uploadUrlAppImage=$1
     uploadUrlUpdate=$2
     echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrlAppImage --header "Content-Type: application/octet-stream" --upload-file $APPIMAGE) 
     echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrlUpdate --header "Content-Type: application/octet-stream" --upload-file $UPDATE) 
 } 

 json=$(upload_release_asset "$uploadUrlAppImage" "$uploadUrlUpdate") 

[antony-jr]

wait no i'm stupid; anthony already pointed it out

desktop/admin/linux/upload-appimage.sh

Lines 26 to 28 in c2f72b5

	export APPIMAGE=$(readlink -f ./Nextcloud*.AppImage)
	export UPDATE=$(readlink -f ./Nextcloud*.AppImage.zsync)
	export BASENAME=$(basename ${APPIMAGE})

desktop/admin/linux/upload-appimage.sh

Line 103 in c2f72b5

uploadUrl=$(echo "${uploadUrl/'{?name,label}'/?name=$BASENAME}")

desktop/admin/linux/upload-appimage.sh

Line 123 in c2f72b5

json=$(upload_release_asset "$uploadUrl")

uploadUrl is only valid for $BASENAME which is based on $APPIMAGE not on $UPDATE

desktop/admin/linux/upload-appimage.sh

Lines 70 to 75 in c2f72b5

	upload_release_asset()
	{
	uploadUrl=$1
	echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl --header "Content-Type: application/octet-stream" --upload-file $APPIMAGE)
	echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl --header "Content-Type: application/octet-stream" --upload-file $UPDATE)
	}

so basically upload_release_asset should take two uploadurls as argument, one for $APPIMAGE and one for $UPDATE
maybe something like this:

 export BASENAME_APPIMAGE=$(basename ${APPIMAGE})
 export BASENAME_UPDATE=$(basename ${UPDATE})

 uploadUrlAppImage=$(echo "${uploadUrl/'{?name,label}'/?name=$BASENAME_APPIMAGE}")
 uploadUrlUpdate=$(echo "${uploadUrl/'{?name,label}'/?name=$BASENAME_UPDATE}")

 upload_release_asset() 
 { 
     uploadUrlAppImage=$1
     uploadUrlUpdate=$2
     echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrlAppImage --header "Content-Type: application/octet-stream" --upload-file $APPIMAGE) 
     echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrlUpdate --header "Content-Type: application/octet-stream" --upload-file $UPDATE) 
 } 

 json=$(upload_release_asset "$uploadUrlAppImage" "$uploadUrlUpdate") 

Yep. That's what I was trying to say without having to write code because I'm too lazy xD. Also it should check for any older .zsync file and delete just like how they delete old AppImage assets just to stay consistent I guess.

[Thatoo]

Could you make a PR @Fuseteam ?

[Fuseteam]

Also it should check for any older .zsync file and delete just like how they delete old AppImage assets just to stay consistent I guess.

ah cool guess i need to search how that works

@Thatoo sure i can do that

[Fuseteam]

@Thatoo done, i just need to look into deleting the old zsync file now

[Thatoo]

Thank you for this PR : https://github.com/nextcloud/desktop/pull/4529/files

Once it's completed with the deletion of old zsync, we'll ask for review.

[Fuseteam]

@Thatoo i've found some remnants of $BASENAME, now i'm less confident if this will work as expected
btw how does it find and delete old appimage files? could you point me in the right direction?

[Thatoo]

In this case, change back $BASENAME_APPIMAGE to $BASENAME and simply add $BASENAME_UPDATE. It is better to add than modify I guess.
About deleting ol appimage, It is in this code :

# Try to delete existing AppImage assets for this PR
assets=$(get_release_assets $releaseId)

for data in $(echo $assets | jq -r '.[] | @uri'); do
    json=$(urldecode "$data")

    assetId=$(echo $json | jq -r '.id')
    name=$(echo $json | jq -r '.name')

    if [[ "$name" == *.AppImage ]]; then
        echo "Deleting old asset: $name"
        $(delete_release_asset $assetId)
    fi
done

I guess you just need to add

    if [[ "$name" == *.AppImage.zsync ]]; then
        echo "Deleting old asset: $name"
        $(delete_release_asset $assetId)
    fi

juste before the "done".

[Fuseteam]

thanks, i'm fairly confident i got all the $BASENAMEs i'm just not confident if this does work as expected:

desktop/admin/linux/upload-appimage.sh

Lines 132 to 133 in 2e80905

	json=$(upload_release_asset "$uploadUrlAppImage" "$uploadUrlUpdate")
	browserDownloadUrl=$(echo $json | jq -r '.browser_download_url')

[Fuseteam]

@Thatoo thanks for the hint, i think this should be it. I've also cleared my doubts now, this should upload the zsync file as intended

[Thatoo]

I have a doubt on

upload_release_asset()
{
    uploadUrl=$1
    asset=$2
    echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrl--header "Content-Type: application/octet-stream" --upload-file $asset)
}

why did you change from

upload_release_asset()
{
    uploadUrlAppImage=$1
    uploadUrlUpdate=$2
    echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrlAppImage --header "Content-Type: application/octet-stream" --upload-file $APPIMAGE)
    echo $(curl --max-time 900 -u $GIT_USERNAME:$GIT_TOKEN -X POST $uploadUrlUpdate --header "Content-Type: application/octet-stream" --upload-file $UPDATE)
}

[Fuseteam]

@Thatoo because as far as i understand we fetch the download url per asset over at

desktop//admin/linux/upload-appimage.sh

Lines 136 to 137 in 2abed2c

	jsonAppImage=$(upload_release_asset "$uploadUrlAppImage" "$APPIMAGE")
	browserDownloadUrlAppImage=$(echo $jsonAppImage | jq -r '.browser_download_url')

[Thatoo]

Ok, got it but did you not you forget a space in the middle of uploadUrl--header ? Should it not be uploadUrl --header.

[Fuseteam]

@Thatoo nice catch i've fixed it

[gjedeer]

Looks like 3.5.4 still doesn't have the zsync information uploaded

[Fuseteam]

@gjedeer it is not merged yet, please wait untill the pull request gets merged

[marc-fouquet]

May I ask why this seems stuck?

[mgallien]

our current tooling for producing AppImage relies on https://github.com/probonopd/linuxdeployqt
that is currently causing a few issues and I have not yet a solution
we will make sure to look at this at the same time

[gjedeer]

@mgallien I asked the linuxdeployqt developer @probonopd if he has any ideas about this problem. From my understanding you're a single command line switch away from getting it to work:

Nextcloud is already embedding update information in the AppImage that gets created by linuxdeployqt, but then that AppImage gets extracted, modified, and recreated without consideration of update information. The --updateinformation argument would need to be used here as per https://github.com/AppImage/AppImageKit#appimagetool-usage.

Finally, the .zsync file would need to be uploaded to https://github.com/nextcloud/desktop/releases just ike the .AppImage and .asc files.

For uploading the zsync file, there is #4529 but it will not work until they also fix the appimagetool invocation to build a new zsync file after modifying the contents of the AppImage.