Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Tvheadend to 4.2.6 (maintenance release) for DSM 5+6 #3050

Merged
merged 7 commits into from
Apr 18, 2018

Conversation

m4tt075
Copy link
Contributor

@m4tt075 m4tt075 commented Dec 17, 2017

Motivation

Update Tvheadend to 4.2.6 (latest maintenance release), update libhdhomerun to 20171221 and adapt Tvheadend to use generic DSM 5+6 service

Linked issues

#2904

Checklist

  • Build rule all-supported completed successfully with 5.2 toolchains
  • Build rule all-supported completed successfully with 6.1 toolchains (exceptions above)
  • Package upgrade completed successfully (on XPE 5.2 and XPE 6.1)
  • New installation of package completed successfully (on XPE 5.2 and XPE 6.1)

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 17, 2017

@ymartin59 Could you take a look at this, please? I think the package is created correctly. It installs fine on XPE5.2, but the service is not executed after the package has been installed. I'm pretty sure that something is wrong with SERVICE_COMMAND, which I have now included in service-setup.sh just like you did for Mosquitto. The -f option should execute tvheadend as daemon, -c defines the home directory for the installation, so no clue why it is looking for a home dir. Another observation: In the install log it says USER=tvheadend, whereas the actually installed user is svc-tvheadend (XPE 5.2), as it should be. Any advice?

EDIT: I cherry-picked your Mosquitto PRs as they seem to contain amendments to the generic service routines. Will drop them from this PR when they have been merged into master...

tvheadend_install.log:

Sun Dec 17 21:42:22 CET 2017
Step preupgrade. USER=tvheadend GROUP= SHARE_PATH=
Sun Dec 17 21:42:22 CET 2017
Step preuninst. USER=tvheadend GROUP= SHARE_PATH=
Sun Dec 17 21:42:22 CET 2017
Step postuninst. USER=tvheadend GROUP= SHARE_PATH=
Sun Dec 17 21:42:22 CET 2017
Step preinst. USER=tvheadend GROUP= SHARE_PATH=
Sun Dec 17 21:42:22 CET 2017
Step postinst. USER=tvheadend GROUP= SHARE_PATH=
Installing service configuration /var/packages/tvheadend/conf/tvheadend.sc
Invoke service_postinst

tvheadend.log:

Sun Dec 17 21:42:27 CET 2017
Starting tvheadend command /volume1/@appstore/tvheadend/bin/tvheadend -f --http_port 9981 --htsp_port 9982 -c /volume1/@appstore/tvheadend/var -p /volume1/@appstore/tvheadend/var/tvheadend.pid
su: can't chdir to home directory '/var/services/homes/svc-tvheadend' 

@m4tt075 m4tt075 changed the title [WIP] Update Tvheadend for DSM5+6 [WIP] Update Tvheadend to 4.2.5 (maintenance release) for DSM 5+6 Dec 18, 2017
@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 18, 2017

@ymartin59 I could track the problem down for now, but still some way to go. Here my observations:

  1. The chdir error appeared because I did not have home directories activated on my XPE5.2 testing system. Apparently that's needed, although I wonder why.
  2. The package can be executed correctly, but only if I hand over the -f (daemon) option and -u svc-tvheadend (user) option and the -g users (group) option to the tvheadend command.
    As username and group vary depending on the DSM version and user/group creation method used, I will have to adapt the tvheadend paramters accordingly. I think you have some variable names for that somewhere in your service scripts, but I still have to find out how I can use them. Work for another day...

@ymartin59
Copy link
Contributor

@m4tt075 Correct, I have configured "home directories" and so have not detected this requirement. As synouser command does not provide a way to change user home path, I think I will call sed to replace default location to package var.

If as I guessed, tvheadend package is designed to run as root and then fork/setuid to specific user, I propose you to adapt previous SSS_SCRIPT to load variables from service-setup and then to provide your own conf/privilege (warning it may be replaced by Makefile ! I have to check that) to declare start/stop operations to be run as root. Probably you also need EFF_USER variable in your script, so I have to set this variable in service-setup too.

I know it is more complex but "generic" support cannot fit all situations. What do you think about this proposal ?

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 19, 2017

@ymartin59 Cool. Thanks. Yes, that would work, but I might have found a different solution: It seems SERVICE_COMMAND is always execute as su ${EFF_USER} -s /bin/sh -c "${SERVICE_COMMAND}" in your scripts. But then I should be able to "reconstruct" EFF_USER and correct group name within service-setup via evaluating id -un and id -gn inside of the SERVICE_COMMAND. Not giving up yet. I will give it a try. If it doesn't work, I will try your suggestion...

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 20, 2017

@ymartin59 I have just pushed some fixes, which utilize a combination of your demoservice concept and the id approach. It works on my XPE 5.2 system. 😃
Could you take a quick look, please, whether this approach makes sense? If so, I'd continue testing on my DSM 6 live system...

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 21, 2017

@ymartin59 I have installed an XPE 6.1 testing system this morning, and the package installs and runs with correct permissions as well. The only problem I'm still facing is that for some reason the sed postinstall commands in the service-setup script are not executed on the 6.1 system, whereas they run fine on the 5.2 system and on 6.1 if the non-generic approach is used.

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 23, 2017

@ymartin59 Yikes! See last commit. Took me hours to find that nasty error. It's the same in your demoservice script, but does not matter there. In any case, from my tests on XPE 5.2 and XPE 6.1 I think everything is working now. I will have to do some more tests (i.e. on my live system) and build all-supported and stuff, but that will have to wait after Christmas...

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 29, 2017

@ymartin59 I have adapted the TVH pacakge using your pre-start method. It installs nicely on my XPE 5.2 and XPE 6.1 test systems. Log-ins work beautifully as well.

There is one issue left: Upgrading the package with itself continues to nuke the old configuration. I have found the to_save_dir variable in your generic installer script, which should actually allow to keep the configuration folder during upgrades. The old configuration should be saved prior to the upgrade and restored after the upgrade. I simply define the variable in the Makefile now, but apparently it is not picked up. I could not find any hints on this in your Wiki documentation either. Can you let me know how to use it correctly and add it in the documentation, please?

Otherwise, the migration should be done. Your review will be appreciated.

@m4tt075
Copy link
Contributor Author

m4tt075 commented Dec 30, 2017

@ymartin59 Contingent to merging of @Safihre 's fixes in PR #3064 , this package now works correctly and upgrades without nuking previous configurations. Without PR #3064 it stays dangerous and should not be merged.

I have cleaned up some additional minors and squashed all changes into a single commit. With this, for your review, please.

@m4tt075 m4tt075 changed the title [WIP] Update Tvheadend to 4.2.5 (maintenance release) for DSM 5+6 Update Tvheadend to 4.2.5 (maintenance release) for DSM 5+6 Jan 2, 2018
@m4tt075
Copy link
Contributor Author

m4tt075 commented Jan 2, 2018

@ymartin59 Minor cleanup in line with your review comments to the analogous Jackett PR #3070 ; squashed and force-pushed. [WIP] label removed from the PR title. From my perspective, ready to go. For final review please. Feel free to merge as you deem appropriate.

Copy link
Contributor

@ymartin59 ymartin59 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. I propose you to add:

echo "service_postinst ${SYNOPKG_PKG_STATUS}" >> $INST_LOG
}

service_preuninst ()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove functions that are useless for your package. I have only written them for demonstration / logging of DSM call sequence.

POST_STRIP_TARGET = tvheadend_extra_install

BUSYBOX_CONFIG = usrmng
ENV += BUSYBOX_CONFIG="$(BUSYBOX_CONFIG)"

include ../../mk/spksrc.spk.mk

.PHONY: tvheadend_install
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this target. In demonstration service it is required as these was no source package with "make install" available.

mkdir --parents $(STAGING_INSTALL_PREFIX)
install -m 755 -d $(STAGING_INSTALL_PREFIX)/bin
install -m 755 src/tvheadend.sh $(STAGING_INSTALL_PREFIX)/bin

.PHONY: tvheadend_extra_install
tvheadend_extra_install:
install -m 755 -d $(STAGING_DIR)/app
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why this additional install invocation is really required as DSM_UI_DIR is expected to trigger standard copy in spksrc makefiles.

HOME_DIR="${SYNOPKG_PKGDEST}/var"
LOG_FILE="${HOME_DIR}/tvheadend.log"
PID_FILE="${HOME_DIR}/tvheadend.pid"
${SYNOPKG_PKGDEST}/bin/tvheadend -f -u ${USRN} -g ${USRG} --http_port ${HTTPP} --htsp_port ${HTSPP} -c ${HOME_DIR} -l ${LOG_FILE} -p ${PID_FILE}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should use $GROUP group which is expected to be sc-media by default

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have just tried to use ${GROUP} instead of ${USRG} but it is an empty variable when used from within service-setup.sh. I will retain ${USRG} for the time being.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to hardcode it as one of first lines in service-setup.sh like @ymartin59 says a bit up here in the other comments and like I did for SABnzbd: 2ac3e53#diff-5d6bd92e6f38c326c2344d53c1dfb20aR10

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, now I see. Misunderstood it. Thanks for clarifying! Will put it in sc-media then...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tried this, but it screws up all the user permissions again. I will revert to the old tvheadend goup and force push to get a cleaner version. I also think you have been reviewing outdated commits, as the tvheadend.sh file has already been removed some days ago.

rm -fr $(STAGING_DIR)
mkdir $(STAGING_DIR)
mkdir --parents $(STAGING_INSTALL_PREFIX)

.PHONY: tvheadend_extra_install
tvheadend_extra_install:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really wonder if this target is really usefull too... With DSM_UI_DIR, folder is copied without such install command.

@ymartin59
Copy link
Contributor

In fact, DSM 6 framework creates service account with a default group with same name - you have used as process argument.
But on DSM 5, no group will be created if GROUP is unset (either from service-setup or from wizard)

@m4tt075
Copy link
Contributor Author

m4tt075 commented Jan 3, 2018

@ymartin59 Thanks. Mostly implemented. See Fixup commit. I'm hesitant w.r.t. the following though:

  • Specifying a download location via the wizard: TVH does not do downloads. There is something like a recording directory. This needs to be specified in the TVH Web-UI. However, I do not see a viable way to "synchronize" the two. It is way easier and more natural for a user installing TVH to select the recording directory via the TVH interface and assign the user permissions afterwards as needed.
  • Enforcing the group-name tvheadend within service-setup.sh by hardcoding: This would match what we have on 6.1 systems. On 5.2 systems the group name should be users though, shouldn't it?

@ymartin59
Copy link
Contributor

OK sorry for my mistake. As DrBean has set "GROUP=sc-media" in installer script, I thought there was a download directory. So forgot about this point about share create and group name. Your shell requesting current user group required as service parameter is perfect so.

@ymartin59
Copy link
Contributor

As a consequence, please clean wizard steps about permission management, as this is not relevant either.

@ymartin59
Copy link
Contributor

... except if tvheadend has to be granted access to other existing share... In that case, GROUP=sc-media will have interesting following effect: it will register EFF_USER as member of that group. So that TVH users can grant service access to existing shares.

@m4tt075
Copy link
Contributor Author

m4tt075 commented Jan 4, 2018

@ymartin59 You are right. I'm hardcoding ${GROUP} to sc-media now, leaving the group permissions for starting tvheadend as daemon untouched. This does allows usage of sc-media to grant permissions for recording directories later on. I have successfully tested this on XPE 6.1 as well as XPE 5.2. So far so good...

However, this leaves me in similar situation to what @Safihre encountered for SABnzbd. During an upgrade process, we have to make sure that existing recording dirs receive read/write permissions for the sc-media group as well. I have been trying, but so far failing, to achieve that. Following @Safihre 's example I try to use the set_syno_permissions function to do it, but it seems to fail on DSM 5.2. Is it supposed to work on 6.1 only? I will push my latest changes, but we are back to WIP, I'm afraid...

@m4tt075 m4tt075 changed the title Update Tvheadend to 4.2.5 (maintenance release) for DSM 5+6 [WIP] Update Tvheadend to 4.2.5 (maintenance release) for DSM 5+6 Jan 4, 2018
@Safihre
Copy link
Contributor

Safihre commented Jan 4, 2018

@m4tt075 I didn't test anything on DSM5.. Do you have an indication what is going wrong in DSM5?
By checking the permissions using synoacltool -get, are any permissions applied?

@m4tt075
Copy link
Contributor Author

m4tt075 commented Jan 4, 2018

@Safihre Nope, I don't. I tried recording but did no have permissions. I have checked the group permission settings in DSM and could verify that they were not set correctly. I checked the install.log for output (and potential errors) from the synoacltool commands and found nothing. I'm running out of ideas...

@ymartin59
Copy link
Contributor

@Safihre we are testing DSM 5 with XPEnology virtual machine... but it is quite complex to set up, and there too proper instructions to succeed at first trial are lacking in wiki.
I will add a "write" test case in demoservice to confirm service process gets permission to write on share.

HTSPP=9982

# Need to enforce correct permissions for recording directories on upgrades
UPGRADE_CFG_DIR="${TMP_DIR}/var/dvr/config"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably it is a better place in service_preupgrade ?

@m4tt075
Copy link
Contributor Author

m4tt075 commented Apr 13, 2018

@ymartin59 @Safihre Thanks, guys, but in the meantime 4.2.6 has been published. In addition the upgrade procedures are dependent on the actual volume number, which can be avoided. I have got the upgrade and the fixes, but still need to compile and push them. Have been drowning recently. Will try to do it over the weekend...

@ymartin59
Copy link
Contributor

No hurry... was just a reminder... backlog is huge !

@m4tt075 m4tt075 changed the title Update Tvheadend to 4.2.5 (maintenance release) for DSM 5+6 Update Tvheadend to 4.2.6 (maintenance release) for DSM 5+6 Apr 13, 2018
@m4tt075
Copy link
Contributor Author

m4tt075 commented Apr 14, 2018

OK, pushed the fixes and the update, adapted PR title and description, travis all green (link), new test builds released (link) and advertised in @Safihre 's test build thread. Since this is the latest release, I'd expect a substantial number of downloads relatively quickly. Let's see how it goes...

@Safihre
Copy link
Contributor

Safihre commented Apr 14, 2018

Unless there's spk specific comments, let's merge and publish by mid next week (18 April). The PR has been open for 4 months..
:)

@Safihre Safihre merged commit ae88485 into SynoCommunity:master Apr 18, 2018
@Safihre
Copy link
Contributor

Safihre commented Apr 18, 2018

No comments received, so time to get this out! Do you have time to build and publish @ymartin59 ? :)

@m4tt075 m4tt075 deleted the tvh-service branch April 19, 2018 05:47
@noncon66
Copy link

Hello,
when I try to install the latest TVheadend (tvheadend_rtd1296-6.1_4.2.6-13.spk) on my DS218play i get this message: "Port undefined configured for this package is either used by another service or reserved for system use. Please disable or modify the conflicting service or contact the developer to modify the package config."

Thanks, noncon66

@m4tt075
Copy link
Contributor Author

m4tt075 commented Apr 21, 2018

Tvheadend should be running on ports 9981 and 9982. Not sure what is blocking it from installing. Any other package that would come to your mind and which you can remove?
If you don't know, you can ssh into your NAS and execute the following commands:
cd /usr/local/etc/services.d
grep -e "ports" *
Look for the package that is using ports 9981 and/or 9982. That's the one that is clashing.

@Safihre
Copy link
Contributor

Safihre commented Apr 22, 2018

@ymartin59 No time to build?

@ymartin59
Copy link
Contributor

@m4tt075 It is possible that previous package revision does not have properly freed port when uninstalling... it happened for me often when testing "broken installer scripts".
@Safihre Sure I will publish it

@m4tt075
Copy link
Contributor Author

m4tt075 commented Apr 27, 2018

@ymartin59 Possible, as there have been several packages from different sources flying around, but I don't believe in some systematic problem. I have had this as well with another package some time ago, but could not reproduce it. The 4.2.4 version and the 4.2.5 and 4.2.6 test builds together must have had thousands of downloads in the meantime. I'm just aware of this single report and the reporter did not come back either.

@noncon66
Copy link

@m4tt075 Sorry for not reporting back. I could not find any other package blocking the port. So I tried a restart. After that the package installed without any problem.

@m4tt075
Copy link
Contributor Author

m4tt075 commented Apr 30, 2018

@ymartin59 I think something went wrong in the publishing process.
https://synocommunity.com/packages still shows Tvheadend 4.2.4-11
https://synocommunity.com/package/tvheadend correctly shows Tvheadend 4.2.6-13, but only the qoriq package. Could you look into this, please?

@ymartin59
Copy link
Contributor

The code of spkrepo really suffers of troubles/limitations/bugs... that new version was not expected to be visible as it was not marked as "active"... I just removed it.
@Safihre @m4tt075 When working on translations, I find out that package relies on sc-media group instead of sc-download. Do you agree to change it so that it matches our "one-group" permissions policy?

@Safihre
Copy link
Contributor

Safihre commented May 2, 2018

This is really a media-package,s o the sc-media makes sense?

@m4tt075
Copy link
Contributor Author

m4tt075 commented May 2, 2018

Yes, that was my understanding as well. It should not interfere with the sc-download packages, which share folders and download locations. Would keep it separate to allow for different permission settings.

stefaang pushed a commit to stefaang/spksrc that referenced this pull request Jan 21, 2019
…unity#3050)

* Update Tvheadend to 4.2.5 (maintenance release) and generic service support for DSM 5+6

* [TVH] Adapt to run as background service and adapt permission setting to new generic scripts

* [TVH] Upgrade fixes: Permissions and handling of potential recordings in package system directory

* [TVH] Improved recording recovery from package root directory

* [TVH] Remove explicit volume dependencies in install scripts for non-standard installations

* [TVH] Update TVH to 4.2.6 (maintenance release)

* [TVH] Remove version patch (no longer required)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants