Backup issue with Python package (is this spksrc related?)

[cthulu]

Hi
since the update to DSM 3.2-1944, I am not able to perform network backup to rsync server. It fails when backing up apps:

Nov 4 21:25:43 PkgMan.cgi: pkgtool.cpp:2198 No script for Python26
Nov 4 21:25:50 synonetbkp: sysbkp_user_export.c:94 fail to write user data
Nov 4 21:25:50 synonetbkp: SYNOSysbkpExport:Fail to export user_desc
Nov 4 21:25:50 synonetbkp: backup_app_apply.c:87 Failed to export application [1], bkptype=[0], szAppPath=[/volume1/@tmp/@app],
synoerr=[0x5500]
Nov 4 21:25:50 synonetbkp: synonetbkp.c:1991 SYNOBackupAppApply() failed!! ulExtension=[1], mode=[MODE_BACKUP], bkptype=
[BACKUP_TYPE_NETBKP], synoerr=[0x5500]

I am not sure whether this is Python package problem (from http://synopkg.superzebulon.org/spkrepo/packages) or something else... More users report this problem, see http://synoblog.superzebulon.org/2011/10/some-news/#comment-8874)

Cheers!

[Diaoul]

It seem Network Backup is trying to backup SPKs. Is this a new behaviour with 1944?

[cthulu]

It seems they added it in 1944. I used to have the Python and SabNZBd
packages on previous version (1932 I think) and no problems there. There
was a minor change further - I started to use zebulon's spk package
repository instead of downloading the files manually. No clue if that
changes anything.
My backup schedule does not include backing up any applications (MySQL etc).

On Mon, Nov 7, 2011 at 9:53 AM, Antoine Bertin <
[email protected]>wrote:

It seem Network Backup is trying to backup SPKs. Is this a new behaviour
with 1944?

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[Diaoul]

Weird.
Do you have only this failure for Python SPK? Can you see other SPKs in the backup log?

The package is correctly installed so the bug comes with DSM 1944 IMO. Did you send a support request to Synology? Maybe they can give us some extra piece of information about that.

[cthulu]

It fails on the first package and does not continue any further, not even
with the backup of normal files :/
I opened a thread on Synology forum, but as this involves 3rd party apps, I
am not sure if there will be much help provided from Synology.

On Mon, Nov 7, 2011 at 10:43 AM, Antoine Bertin <
[email protected]>wrote:

Weird.
Do you have only this failure for Python SPK? Can you see other SPKs in
the backup log?

The package is correctly installed so the bug comes with DSM 1944 IMO. Did
you send a support request to Synology? Maybe they can give us some extra
piece of information about that.

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[Diaoul]

Well, 3rd party apps are supported in DSM through the use of SPKs. If a normal/conform SPK makes Synology features fail, it's also Synology's problem.

Here you can fill a support form: http://www.synology.com/support/support_form.php?lang=enu
You'll get an answer from Synology for sure in usually 3 working days.

[cthulu]

OK, I will fill in support form and let you know. I thought it might be
something easy to fix, documented in the SPK format.
I have no way of testing if this is present in older release. Can someone
test if this is not an issue in 3.2-1922?
Reproduce:

1. install Python 2.6 spk
2. setup network backup (another rsync server, for example to
   strongspace.com, maybe also local rsync server)
3. execute network backup task

On Mon, Nov 7, 2011 at 11:19 AM, Antoine Bertin <
[email protected]>wrote:

Well, 3rd party apps are supported in DSM through the use of SPKs. If a
normal/conform SPK makes Synology features fail, it's also Synology's
problem.

Here you can fill a support form:
http://www.synology.com/support/support_form.php?lang=enu
You'll get an answer from Synology for sure in usually 3 working days.

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[zebulon501]

I try to reproduce this issue on my side, and here are my findings:

• On first try, I got the same issue, so it seems consistent
• I added a start-stop-status script (see first line of your report), but this doesn't change anything
• I uninstalled Python 2.6 completely, doesn't help
• I removed all packages, and the backup was working
• I reinstalled Python 2.6, backup is still working
• I reinstalled all the other packages, and it still works

So I think it is related to the way the users are created. These users are created using the adduser command (from busybox), but in the user range, not the system range (uid > 1000). This seems to trouble the DSM software: on the next reboot, these users are create in the DSM layer (they appear in the DSM interface, but they shouldn't as they are service users). I'll try to investigate more on this and keep you posted.

[Diaoul]

The best would be to create system users as it is done on standard Linux OS.
Apache2 runs with user www-data on Debian. What is the software behind the creation of system user www-data on apache2 installation?

From man adduser:

-r, --system
Create a system account.

System users will be created with no aging information in 
/etc/shadow, and their numeric identifiers are choosen in
the SYS_UID_MIN-SYS_UID_MAX range, defined in /etc/login.defs,
instead of UID_MIN-UID_MAX (and their GID counterparts for the creation of groups).

Note that useradd will not create a home directory for such an user,
regardless of the default setting in /etc/login.defs (CREATE_HOME).
You have to specify the -m options if you want a home directory for a
system account to be created. 

Maybe that could be useful

Edit: Except I don't know if that feature is available with busybox' version of adduser...

[zebulon501]

busybox's adduser have a -S switch which does pretty the same. The issue here is for existing installations, were the user id will change, and this might lead to permissions issues, and a whole bunch of questions like were is the XXX user gone, I don't see it in the DSM interface any more, how can I be sure it doesn't create a security hole in my system as I can't see it, and so on. For me, these are the main issues in changing the way these system users: migration of current installation and communication with users.

[Diaoul]

So the question is, what to do at uninstallation time to clean up files created by the user we're about to delete.

Maybe a piece of answer in man userdel

userdel - Delete a user account and related files

Delete related files. That seem the best way to do it.
However, we have to way to warn the user about that when he is uninstalling a package. And he won't expect such a behavior and will realize his files are gone without any notice.

Another solution might be to create a "clean up" SPK that will scan the system for orphelon files (in data partition only, ie ^/volume[0-9]+/[^@].*) and give them a new user (for example, current logged on user in DSM by default, but customizable through an interface).

[cthulu]

Are we sure the problem is the way users are created? The backup was
working before 3.2-1944 and it seems a lot of complexity added by Synology
for such a small "hotfix" release.

On Mon, Nov 7, 2011 at 3:40 PM, Antoine Bertin <
[email protected]>wrote:

So the question is, what to do at uninstallation time to clean up files
created by the user we're about to delete.

Maybe a piece of answer in man userdel

userdel - Delete a user account and related files

Delete related files. That seem the best way to do it.
However, we have to way to warn the user about that when he is
uninstalling a package. And he won't expect such a behavior and will
realize his files are gone without any notice.

Another solution might be to create a "clean up" SPK that will scan the
system for orphelon files (in data partition only, ie
^/volume[0-9]+/[^@].* and give them a new user (for example, current
logged on user in DSM by default, but customizable through an interface).

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[Diaoul]

IMO we share responsibilities with Synology:

• The way user are managed isn't perfect in SPKs
• Synology may have introduced some extra stuff that causes trouble when created users aren't perfect

[cthulu]

I have submitted issue to Synology, in case more info is needed.

[cthulu]

Response from Synology:

Dear Matej,

Thank you for your message.

We do not support the problem that caused by third party package. As you've mentioned that the backup works when you uninstall the package. We do not responsible for any third party package.

If you have further questions or suggestions, please feel free to contact us again.

Best regards,
-Adrian Hsieh

[Diaoul]

When it's time to help community, Synology is often missing, I asked for up-to-date documentation of their guide when they released 3.2, two months ago. Still nothing done.
I guess we can't expect any help from them.

[cthulu]

Guys, any news on this issue? Or workaround? Like manually creating users or so...

[Diaoul]

Seems uninstalling all packages and reinstalling everithing is a workaround

[cthulu]

Hm I can try it. What happens with the databases and settings? Also, should
you or shouldn't you reboot after that?
On Nov 12, 2011 4:12 PM, "Antoine Bertin" <
[email protected]>
wrote:

Seems uninstalling all packages and reinstalling everithing is a workaround

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[Diaoul]

All gone. Plus it may not be recommended because there is no user clean-up package yet so you'll still have some files that belongs to a user that no longer exists.

The best is to wait for now I think.

[zebulon501]

No risk of leftover files.

[Diaoul]

And what about downloaded files with SABnzbd?

[cthulu]

So, I've uninstalled all packages and network backup was working. I manually deleted users (sickbeard etc). After installing Python package, the backup stopped working again.
Engineer from Synology remotely logged to my box, but haven't heard anything from them since them. I sent them mail couple of times asking about progress, but no response followed...

[Diaoul]

Check in /etc/passwd to see if there are still some users remaining (sickbeard, etc.)
As python package doesn't create any user, I don't think it is related to it.
Also, you can check in /usr/local if there is some stuff remaining. There shouldn't if all packages are uninstalled.

[Diaoul]

Is this happening only with python SPK ?

[cthulu]

i checked passwd for extra users, none present.
i couldn't check with other packages, because they all depend on python
when i uninstalled python, everything was working again, installing it
again via package repo re-created the problem :(
in the log this appears: synonetbkp: netbkp_rsync_execv.c:103 Failed to
execute rsync command.

On Sat, Nov 26, 2011 at 6:29 PM, Antoine Bertin <
[email protected]

wrote:

Is this happening only with python SPK ?

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[Diaoul]

Try http://syno.diaoul.fr/SPKs/umurmur/
No repo but you still can install the SPK the old way

[cthulu]

I've tried your package, it is working. Also, when I removed all packages,
cleaned up passwd, python package alone seemed to be working. After
installation of sab/sickbeard, the problems started again. I tried
installing the old packages (before repo) and the problem persisted, even
when I rebooted between installs. I unfortunately cannot spend more time
un- and reinstalling the packages, maybe it'll be helpful if someone else
tried it as well as I may miss something...
On Nov 26, 2011 7:58 PM, "Antoine Bertin" <
[email protected]>
wrote:

Try http://syno.diaoul.fr/SPKs/umurmur/
No repo but you still can install the SPK the old way

---

Reply to this email directly or view it on GitHub:
#14 (comment)

[cthulu]

Just a quick one - the issue is still present in 3.2-1955 releaesed on 30-nov.

[cthulu]

Reply from Synology (after investigation remotely on my system):

A bug was created on our bug system to record this issue. The problem happened when system trying to enumerate users. However, the problem only happened when user installed 3rd party software.

We may fix the problem in the future.

So nothing solved :(

[Diaoul]

If we cannot create system users this is gonna be a real problem for us.

Wait and see...

[Diaoul]

Might be solved using -S option when creating users in SPKs. An system uid is given to the user. It should not conflict with DSM anymore.

I do that in nasforum branch, needs testing but I think this is the most we can do here.

[cthulu]

Can someone verify if this is the fix? Or is this fixed in DSM 4.0?

[Diaoul]

I will check on my branch but I don't know for superzebulon's. My uids are in the system range now, I think this might solve the issue.