Support for sync-collection REPORT

[oupala]

After discussing with the author of CalDavZAP, CardDavMATE and InfCloud, I think that Radicale lacks an important feature : support for sync-collection REPORT (with sync-tokens).

This is how InfCloud react when a server does not support the sync-collection REPORT feature :

• do
  • download a list of ALL URLs+Etags for all objects => bandwidth consuming
  • if there is an URL not already downloaded then this is a new object, then download it => cpu consuming
  • if there is an URL which is already downloaded but the Etag is changed, this object was updated, then re-download it => cpu consuming
  • if there is an URL which is already downloaded but no more present on the server, this object was deleted, then delete it => cpu consuming
• while(true);

This is how InfCloud react when a server does support the sync-collection REPORT feature (such as DAViCal) :

• download a list of ALL URLs+Etags + get the sync-token => bandwidth consuming but only once
• do
  • perform the synchronization request with the last sync-token => minimum bandwith usage
  • download all new objects returned by the request (if there is any) => nearly zero cpu usage
  • re-download all changed objects returned by the request (if there is any) => nearly zero cpu usage
  • delete all deleted objects returned by the request (if there is any) => nearly zero cpu usage
• while(true);

It could be a good idea to implement this feature.

[midemarc]

+1, I see it as a must-have feature !

[Unrud]

The response of a propfind request for all etags of a calendar collection with 1000 calendar objects is about 240kB. The response size scales pretty much linear with the number of calendar objects.
gzip reduces the size by a factor of about 10. Maybe the integrated server should support his.

For sync-token support the storage backends need to keep track of all calendar object's time of last change / time of deletion.

[liZe]

For sync-token support the storage backends need to keep track of all calendar object's time of last change / time of deletion.

I've come to the same conclusion, and that's a bit hard to add. I change the milestone from 2.0 to later, I'd prefer to have more time to implement this.

[Unrud]

It's also possible to use the collection etag as sync-token. When the etag has changed, the server just tells the client that the sync-token is invalid (see RFC). The client falls back to a normal report request.

As long as the collections is not changed very often, this should save nearly as much bandwidth as a real sync-token implementation.

[pbiering]

It looks like this missing sync-token implementation (even a simple one) will break vcard sync with SoGo connector, I have here various vcard update pending in Thunderbird, but they are not pushed to server. See also https://sogo.nu/bugs/view.php?id=3896 (looks like the connector has no fallback in case the sync-token is not provided).

REPORT request found with DEPTH=1

<?xml version="1.0"?><sync-collection xmlns="DAV:"><sync-token/><prop><getetag/><getcontenttype/></prop></sync-collection>

Answer:

<?xml version="1.0"?>
<multistatus xmlns="DAV:">
  <response>
    <href>/URI/UUID.vcf</href>
    <propstat>
      <prop>
        <getetag>"2229d62987e2e5c15c04a6ca52f1b8cd"</getetag>
        <getcontenttype>text/vcard; component=vcard</getcontenttype>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

How difficult is it to implement the simple way Unrud suggested on 13 Jul? Can one point me to the related code option or create a pull request on top of "pbiering-testing" and I will test the code then here on my system.

Thank you!

[pbiering]

it turned out that PROPFIND at least since 0.10 offers

          <supported-report>
            <report>
              <sync-collection />
            </report>
          </supported-report>

but no real support is implemented behind currently. Removed the offering, now the sync is working again with SOGo connector (which activates fallback then).

Related code is in xmlutils.py:

            for report_name in (
                    "principal-property-search", "sync-collection",
                    "expand-property", "principal-search-property-set"):
                supported = ET.Element(_tag("D", "supported-report"))

See commit on "pbiering-testing":
2bd1b4f

[oupala]

Is it implemented?

[Unrud]

Is it implemented?

Yes

[oupala]

Great news. It was not obvious for my eyes if it was implemented or not. Thanks for the confirmation.