unreadable folder in external smb share raises unhandled ForbiddenException during occ files:scan

[derhedwig]

Hi!

Summary

occ files:scan raises ForbiddenException when I scan an external smb share that includes a folder to which the user has no access to. The same seems to happen when the cron job runs.

Stack trace

$ sudo -u apache php /var/www/nextcloud/occ files:scan
Starting scan for user 1 out of 1 (redacted)
Exception during scan: Invalid request for /Private (ForbiddenException)
#0 /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Native/NativeState.php(66): Icewind\SMB\Exception\Exception::fromMap(Array, 13, '/Private')
#1 /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Native/NativeState.php(78): Icewind\SMB\Native\NativeState->handleError('/Private')
#2 /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Native/NativeState.php(109): Icewind\SMB\Native\NativeState->testResult(false, 'smb://smb.example.com')
#3 /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Native/NativeShare.php(92): Icewind\SMB\Native\NativeState->opendir('smb://smb.example.com')
#4 /var/www/nextcloud/apps/files_external/lib/Lib/Storage/SMB.php(236): Icewind\SMB\Native\NativeShare->dir('Private')
#5 /var/www/nextcloud/apps/files_external/lib/Lib/Storage/SMB.php(604): OCA\Files_External\Lib\Storage\SMB->getFolderContents('Private')
#6 [internal function]: OCA\Files_External\Lib\Storage\SMB->getDirectoryContent('Private/Private')
#7 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(411): iterator_to_array(Object(Generator))
#8 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(388): OC\Files\Cache\Scanner->handleChildren('Private/Private...', true, 3, 385, true, 0)
#9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->scanChildren('Private/Private...', true, 3, 385, true)
#10 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->scanChildren('Private', true, 3, 377, true)
#11 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(340): OC\Files\Cache\Scanner->scanChildren('', true, 3, 371, true)
#12 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(260): OC\Files\Cache\Scanner->scan('', true, 3)
#13 /var/www/nextcloud/apps/files/lib/Command/Scan.php(151): OC\Files\Utils\Scanner->scan('/hedwig', true, NULL)
#14 /var/www/nextcloud/apps/files/lib/Command/Scan.php(207): OCA\Files\Command\Scan->scanFiles('hedwig', '/hedwig', Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#15 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/nextcloud/core/Command/Base.php(169): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1012): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /var/www/nextcloud/3rdparty/symfony/console/Application.php(272): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /var/www/nextcloud/3rdparty/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#20 /var/www/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#21 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#22 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#23 {main}

Possible solution

I found a solution, simply by enclosing the relevant function call in a try catch block:

--- /tmp/SMB.php	2020-08-19 00:37:12.773360812 +0200
+++ /var/www/nextcloud/apps/files_external/lib/Lib/Storage/SMB.php	2020-08-18 23:59:51.567888500 +0200
@@ -233,7 +233,11 @@
 	protected function getFolderContents($path): iterable {
 		try {
 			$path = ltrim($this->buildPath($path), '/');
-			$files = $this->share->dir($path);
+			try {
+				$files = $this->share->dir($path);
+			} catch (ForbiddenException $e) {
+				$files = [];
+			}
 			foreach ($files as $file) {
 				$this->statCache[$path . '/' . $file->getName()] = $file;
 			}

Steps to reproduce

1. Setup an SMB share via external_files.
2. Turn check_acl off.
3. Have folders on the share, the user does not have access to.
4. run occ files:scan --all.

Expected behaviour

The scan should complete, ignoring the folders that cannot be read.

Actual behaviour

The scan crashes at the unreadable folder.

Server configuration

Operating system: CentOS Linux release 8.2.2004 (Core)

Web server: Apache/2.4.37 (centos)

Database: mysql Ver 15.1 Distrib 10.3.17-MariaDB, for Linux (x86_64) using readline 5.1

PHP version: PHP 7.4.9

Nextcloud version: 19.0.1

Updated from an older Nextcloud/ownCloud or fresh install: fresh install

Where did you install Nextcloud from: https://download.nextcloud.com/server/releases/latest.tar.bz2

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.5.0
  - activity: 2.12.0
  - admin_audit: 1.9.0
  - bruteforcesettings: 1.6.0
  - calendar: 2.0.3
  - cloud_federation_api: 1.2.0
  - comments: 1.9.0
  - contacts: 3.3.0
  - contactsinteraction: 1.0.0
  - dav: 1.15.0
  - deck: 1.0.5
  - documentserver_community: 0.1.7
  - federatedfilesharing: 1.9.0
  - federation: 1.9.0
  - files: 1.14.0
  - files_external: 1.10.0
  - files_pdfviewer: 1.8.0
  - files_rightclick: 0.16.0
  - files_sharing: 1.11.0
  - files_trashbin: 1.9.0
  - files_versions: 1.12.0
  - files_videoplayer: 1.8.0
  - firstrunwizard: 2.8.0
  - logreader: 2.4.0
  - lookup_server_connector: 1.7.0
  - mail: 1.4.1
  - nextcloud_announcements: 1.8.0
  - notifications: 2.7.0
  - oauth2: 1.7.0
  - onlyoffice: 4.3.0
  - password_policy: 1.9.1
  - privacy: 1.3.0
  - provisioning_api: 1.9.0
  - serverinfo: 1.9.0
  - settings: 1.1.0
  - sharebymail: 1.9.0
  - support: 1.2.1
  - survey_client: 1.7.0
  - systemtags: 1.9.0
  - text: 3.0.1
  - theming: 1.10.0
  - twofactor_backupcodes: 1.8.0
  - twofactor_totp: 4.1.3
  - updatenotification: 1.9.0
  - user_ldap: 1.9.0
  - viewer: 1.3.0
  - workflowengine: 2.1.0
Disabled:
  - encryption
  - photos
  - recommendations

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "**REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "19.0.1.1",
        "overwrite.cli.url": "**REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": true,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "default_language": "de",
        "default_locale": "de_DE",
        "allow_user_to_change_display_name": false,
        "skeletondirectory": false,
        "lost_password_link": "disabled",
        "logfile": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "htaccess.RewriteBase": "\/",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": []
    }
}

Are you using external storage, if yes which one: smb

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP

LDAP configuration (delete this part if not used)

LDAP config

+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Configuration                 | s01                                                                                                                               |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport      | 1                                                                                                                                 |
| homeFolderNamingRule          |                                                                                                                                   |
| lastJpegPhotoLookup           | 0                                                                                                                                 |
| ldapAgentName                 | uid=nextcloud,cn=sysaccounts,cn=etc,dc=ipa,dc=janun,dc=de                                                                         |
| ldapAgentPassword             | ***                                                                                                                               |
| ldapAttributesForGroupSearch  |                                                                                                                                   |
| ldapAttributesForUserSearch   | displayName;mail;uid                                                                                                              |
| ldapBackupHost                |                                                                                                                                   |
| ldapBackupPort                |                                                                                                                                   |
| ldapBase                      | dc=ipa,dc=janun,dc=de                                                                                                             |
| ldapBaseGroups                | cn=groups,cn=accounts,dc=ipa,dc=janun,dc=de                                                                                       |
| ldapBaseUsers                 | cn=users,cn=accounts,dc=ipa,dc=janun,dc=de                                                                                        |
| ldapCacheTTL                  | 600                                                                                                                               |
| ldapConfigurationActive       | 1                                                                                                                                 |
| ldapDefaultPPolicyDN          |                                                                                                                                   |
| ldapDynamicGroupMemberURL     |                                                                                                                                   |
| ldapEmailAttribute            | mail                                                                                                                              |
| ldapExperiencedAdmin          | 0                                                                                                                                 |
| ldapExpertUUIDGroupAttr       | ipaUniqueID                                                                                                                       |
| ldapExpertUUIDUserAttr        | ipaUniqueID                                                                                                                       |
| ldapExpertUsernameAttr        | uid                                                                                                                               |
| ldapExtStorageHomeAttribute   |                                                                                                                                   |
| ldapGidNumber                 | gidNumber                                                                                                                         |
| ldapGroupDisplayName          | cn                                                                                                                                |
| ldapGroupFilter               | (&(objectclass=groupofnames)(!(cn=admins))(!(cn=editors))(!(cn=ipausers))(!(cn=useradmins))(!(cn=trust admins))(!(cn=nextcloud))) |
| ldapGroupFilterGroups         |                                                                                                                                   |
| ldapGroupFilterMode           | 1                                                                                                                                 |
| ldapGroupFilterObjectclass    |                                                                                                                                   |
| ldapGroupMemberAssocAttr      | member                                                                                                                            |
| ldapHost                      | redacted                                                                                                                 |
| ldapIgnoreNamingRules         |                                                                                                                                   |
| ldapLoginFilter               | (&(&(objectclass=*)(memberof=cn=nextcloud,cn=groups,cn=accounts,dc=ipa,dc=janun,dc=de))(uid=%uid))                                |
| ldapLoginFilterAttributes     |                                                                                                                                   |
| ldapLoginFilterEmail          | 0                                                                                                                                 |
| ldapLoginFilterMode           | 1                                                                                                                                 |
| ldapLoginFilterUsername       | 1                                                                                                                                 |
| ldapNestedGroups              | 1                                                                                                                                 |
| ldapOverrideMainServer        |                                                                                                                                   |
| ldapPagingSize                | 500                                                                                                                               |
| ldapPort                      | 389                                                                                                                               |
| ldapQuotaAttribute            |                                                                                                                                   |
| ldapQuotaDefault              |                                                                                                                                   |
| ldapTLS                       | 0                                                                                                                                 |
| ldapUserAvatarRule            | default                                                                                                                           |
| ldapUserDisplayName           | displayname                                                                                                                       |
| ldapUserDisplayName2          |                                                                                                                                   |
| ldapUserFilter                | (&(objectclass=*)(memberof=cn=nextcloud,cn=groups,cn=accounts,dc=ipa,dc=janun,dc=de))                                             |
| ldapUserFilterGroups          |                                                                                                                                   |
| ldapUserFilterMode            | 1                                                                                                                                 |
| ldapUserFilterObjectclass     |                                                                                                                                   |
| ldapUuidGroupAttribute        | auto                                                                                                                              |
| ldapUuidUserAttribute         | auto                                                                                                                              |
| turnOffCertCheck              | 0                                                                                                                                 |
| turnOnPasswordChange          | 0                                                                                                                                 |
| useMemberOfToDetectMembership | 1                                                                                                                                 |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+

Client configuration

Browser: Chrome 84.0.4147.89

Operating system: Fedora release 32 (Thirty Two)

Logs

Web server error log

Web server error log

Nothing relevant found

[derhedwig]

This is similiar to #9758, but #9608 did obviously not fix this.

[derhedwig]

Hm. I'm still getting ForbiddenException even with check_acl turned on.

[ce-data]

We had the same issue here which rendered our NC instance kind of unusable, because as a consequence, a lot of files were not synced by the desktop clients.

Enabling ACLs improved the situation (it does not crash on every unreadable directory), but it still crashed on directories where the user only has traverse permissions (i.e. Linux permissions: "--x").

After adding the try/catch-block, everying worked again.

Thanks a lot!

Hopefully, it will be fixed in the next release.

BTW: I'm on 19.0.2 right now. I haven't tried 19.0.3, yet, but the SMB.php still looks the same:
https://github.com/nextcloud/server/commits/master/apps/files_external/lib/Lib/Storage/SMB.php

[ce-data]

This issue still exists in 20.0.8:

Exception during scan:
#0 /daten/nextcloud/apps/files_external/lib/Lib/Storage/SMB.php(615): OCA\Files_External\Lib\Storage\SMB->getFolderContents('DIRECTORYNAME')
#1 [internal function]: OCA\Files_External\Lib\Storage\SMB->getDirectoryContent('DIRECTORYNAME')
#2 /daten/nextcloud/lib/private/Files/Cache/Scanner.php(408): iterator_to_array(Object(Generator))
#3 /daten/nextcloud/lib/private/Files/Cache/Scanner.php(388): OC\Files\Cache\Scanner->handleChildren('DIRECTORYNAME', true, 3, 9558660, true, 0)
#4 /daten/nextcloud/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->scanChildren('DIRECTORYNAME', true, 3, 9558660, true)
#5 /daten/nextcloud/lib/private/Files/Cache/Scanner.php(340): OC\Files\Cache\Scanner->scanChildren('', true, 3, 9558659, true)
#6 /daten/nextcloud/lib/private/Files/Utils/Scanner.php(260): OC\Files\Cache\Scanner->scan('', true, 3)
#7 /daten/nextcloud/apps/files/lib/Command/Scan.php(152): OC\Files\Utils\Scanner->scan('/06129A41-DE4F-...', true, NULL)
#8 /daten/nextcloud/apps/files/lib/Command/Scan.php(208): OCA\Files\Command\Scan->scanFiles('06129A41-DE4F-4...', '/06129A41-DE4F-...', Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#9 /daten/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /daten/nextcloud/core/Command/Base.php(169): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /daten/nextcloud/3rdparty/symfony/console/Application.php(1000): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /daten/nextcloud/3rdparty/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /daten/nextcloud/3rdparty/symfony/console/Application.php(147): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /daten/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /daten/nextcloud/console.php(100): OC\Console\Application->run()
#16 /daten/nextcloud/occ(11): require_once('/daten/nextclou...')
#17 {main}

Since commit e9ae943 changed the SMB.php, I moved the try catch block to Scanner.php:

--- /daten/nextcloud/lib/private/Files/Cache/Scanner.php.orig   2021-03-04 22:31:16.138907908 +0100
+++ /daten/nextcloud/lib/private/Files/Cache/Scanner.php        2021-03-04 22:37:53.635728117 +0100
@@ -405,8 +405,11 @@
        private function handleChildren($path, $recursive, $reuse, $folderId, $lock, &$size) {
                // we put this in it's own function so it cleans up the memory before we start recursing
                $existingChildren = $this->getExistingChildren($folderId);
-               $newChildren = iterator_to_array($this->storage->getDirectoryContent($path));
-
+               try {
+                       $newChildren = iterator_to_array($this->storage->getDirectoryContent($path));
+                } catch (\OCP\Files\NotPermittedException $e) {
+                       $newChildren = [];
+               }
                if ($this->useTransactions) {
                        \OC::$server->getDatabaseConnection()->beginTransaction();
                }

[szaimen]

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

[ghost]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.