File Save As... fails in Firefox (but succeeds when uMatrix is disabled)

[bhrgunatha]

Prerequisites

• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
  • I also searched the existing issues at https://github.com/gorhill/uMatrix/issues
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uMatrix
• I tried to reproduce the issue when...
  • uMatrix is the only extension
  • uMatrix with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uMatrix
• I checked the documentation to understand that the issue I report is not a normal behavior
• I used the logger to rule out that the issue is caused by my ruleset

Description

A specific URL where the issue occurs

https://pastebin.com/HjWANtHk

Steps to Reproduce

1. Visit some site - e.g. pastebin above but it also happens when trying to save json from reddit.
2. Click raw - the text only file appears.
3. File | Save Page As... (or CTRL-S)
4. Choose any location.
5. Click Save.
6. Go to Downloads.
7. See file has Failed.
8. Click the retry icon (right hand side of downloads list).
9. File saves correctly.

Similar example

1. Visit my saved links on reddit - https://www.reddit.com/user/bhrgunatha/saved#links

2. Change the url suffix saved#links -> saved.json#links to receive json instead of html - https://www.reddit.com/user/bhrgunatha/saved.json#links

3. File Save As...

4. Steps same as above.

Ruleset

Active rules

Supporting evidence

Logger output

Your environment

• uMatrix version: 1.3.14
• Browser Name and version: Firefox 64.0
• Operating System and version: Arch linux - kernel - 4.19.8 x86_64

[uBlock-user]

Other column is blocked, the request is tabless and identified in other cell column which is blocked.

[gorhill]

I can reproduce the page not saving, but I cannot reproduce an actual issue. The logger shows a behind-the-scene network request being blocked, hence it's a user ruleset issue, not a uMatrix issue. With the new UI in 1.3.15rc1, I can set a rule pastebin.com 1st-party other allow from the logger and saving then work as expected.

[bhrgunatha]

My global rules:

https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: edu.swi-prolog.org true
matrix-off: localhost true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: wyciwyg-scheme true
referrer-spoof: behind-the-scene false
* * css allow
* * image allow
* cdnjs.cloudflare.com script allow
* code.jquery.com script allow
* d3js.org script allow
* gcs-vimeo.akamaized.net media allow
* googlevideo.com media allow
* player.vimeo.com frame allow
* player.vimeo.com other allow
* player.vimeo.com script allow
* vimeocdn.com frame allow
* vimeocdn.com media allow
* vimeocdn.com other allow
* vimeocdn.com script allow
* www.youtube.com script allow
* youtube.com frame allow

I don't have any rules for pastebin.

Are you saying in order to use the browser's built in File Save As with umatrix 'other' must be allowed either for specific sites or globally?

Sorry I don't understand where the behind the scene blocking is coming from in the logger - only those 2 cookies.

[gorhill]

In the logger, select "All" or "Behind the scene" and see if you can see behind-the-scene entries.

In any case, the stable version won't allow you to create a rule from that entry in the logger, this has been addressed in 1.3.15rc1, which I should release hopefully soon (but now I need to revert the migration to Fontawesome 5, plus other stuff...)

[uBlock-user]

Add matrix-off: behind-the-scene true and try to reproduce again.

[gorhill]

@uBlock-user It's not going to work, the context is pastebin.com not behind-the-scene. This is a tabless network request, but with a valid context. This is what is fixed in 1.13.15rc1, you can set rules in the proper context instead of the behind-the-scene one.

[bhrgunatha]

Both All and behind-the-scene are blank in the logger - also after adding the matrix-off rule.

However I now see a 1 in the 'other' column - and if I allow that I can save (as usual.)

I don't understand what 'other' really represents though. Can websites use/abuse 'other' network requests?

[uBlock-user]

Can it still be a tabless request if it has a valid context ? I always understood that if it is BTS, it doesn't have a valid context.

[uBlock-user]

I now see a 1 in the 'other' column - and if I allow that I can save (as usual.)

you should whitelist other column for first-party domains.

I don't understand what 'other' really represents though.

Other represents what is not classified under scripts,images,XHR,css, media, cookie, frame etc..

[gorhill]

I don't understand what 'other' really represents though.

https://github.com/gorhill/uMatrix/wiki/The-popup-panel#the-type-cells

[gorhill]

Can it still be a tabless request if it has a valid context ?

Yes. Originally behind-the-scene meant no tab/no context. With Chromium 63 and Firefox Webext, it's possible a network request is not associated with a tab (extenstions framework says tabId === -1), but now it can have context (initiator for Chromium, documentUrl for Firefox) even though there is still no tab. Hence now I use "tab-less" and context-less. The later should be quite rare now, but if ever it occurs, it will be filtered with the context behind-the-scene, that is the only time that special context is used now.

[gorhill]

What's to address?

All worked as expected on my side, the logger reported everything, just a matter of selecting All to see everything. The only quirk is to set rule on a tab-less request and this was fixed in gorhill/uMatrix#951 (comment).

[gwarser]

Something wrong with Firefox. It insists to re-download already visible content when saving, download fails because is blocked, then succeeds on retry. Extensions logger does not show this retry attempt, but it is visible in browser console.

Share your thoughts here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1343466

[uBlock-user]

What's to address?

So I will close it then, you kept it open and didn't close it, so I thought you had to address something.

[uBlock-user]

I could only reproduce the second example of reddit link, which is now forbidden, but didn't happen after I whitelisted tabless request.

[gorhill]

Share your thoughts here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1343466

Seems to be a UI thing in Firefox. The page was properly saved on my side, except for the blocked items. So it seems Firefox is marking the download as failed just because some items were blocked by uBO:

[d-g]

Seems to be a UI thing in Firefox. The page was properly saved on my side, except for the blocked items. So it seems Firefox is marking the download as failed...

As for me, no, the download actually fails. I’ve just encountered this bug using µMatrix 1.3.17b1 after updating Firefox from 60.4.0 ESR to 64.0.2 (current release), so this is clearly a regression.

With FF60 saving works well with µMatrix disabled ‘behind-the-scene’ per default. In FF64 when trying to save any page I only get ‘Failed’ and a bunch of lines like these on the logger console:

14:54:21	www.gnu.org	--	other	https://www.gnu.org/graphics/fs-gang.png
14:54:21	www.gnu.org	--	other	https://www.gnu.org/print.min.css
...

None of these files are saved despite that they are loaded from the net and shown on a page: I only got *.html and empty *_files directory.

Rules are following:

https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: behind-the-scene true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: vivaldi-scheme true
matrix-off: wyciwyg-scheme true
noscript-spoof: * true
referrer-spoof: * true
referrer-spoof: behind-the-scene false
* * * block
* * css allow
* * frame allow
* * image allow

All other settings are default.

As far as I understood, the only thing could be done is to allow ‘other’ globally, i. e.:

* * other allow

Is it indeed so? If yes, this ‘other’ switch becomes nearly unusable in new Firefox versions, as saving pages is crucial feature of a browser.

What's to address? All worked as expected on my side

In the light of the above, it would be very nice of you, if users were at least prominently notified about such breaking changes even when things have been broken by Mozilla.

[gorhill]

I only get ‘Failed’ and a bunch of lines like these on the logger console

So you got your answer, you need to allow other for www.gnu.org -- you can do this from the logger.

This is on the front page of the uMatrix project (emphasis in original):

Using uMatrix logger is key to un-break sites: the logger will show you all that uMatrix does internally.

[gorhill]

Alright, I am going to give another look at what can be done, if anything. The tablessness issue has always been a thorny one, the is no obvious correct solution.

In the previous version, uMatrix was attempting to find a tab matching the contextual URL of the request, and if found would use this tab to report the network request. This was put in place strictly to provide a temporary mean to set proper rule until the logger itself had the built-in ability to set such rule.

Once the logger gained that ability, I removed that temporary solution. Given that one context URL may match more than one tab, a proper solution would be to report to all the matching tabs. Whether this is a feasible solution in practice is not clear.

[h1z1]

Not sure this is entirely Firefox unique as I run into something very similar with Chrome* browsers too. Simple example is you can't save a pdf from within pdfjs unless you allow Other even though you're already viewing it. I understand why it happens and know what "other" means in this context but it's not obvious. The save process appears successfull.

Example

Rules

    https-strict: behind-the-scene false
    matrix-off: about-scheme true
    matrix-off: behind-the-scene true
    matrix-off: chrome-extension-scheme true
    matrix-off: chrome-scheme true
    matrix-off: moz-extension-scheme true
    matrix-off: opera-scheme true
    matrix-off: wyciwyg-scheme true
    noscript-spoof: * true
    referrer-spoof: * true
    referrer-spoof: behind-the-scene false
    * * * block
    * 1st-party cookie block
    * 1st-party css block
    * 1st-party frame block
    * 1st-party image block
    * 1st-party media block
    * 1st-party other block
    * 1st-party script block
    * 1st-party xhr block
    xiph.org people.xiph.org css allow
    xiph.org people.xiph.org image allow
    xiph.org people.xiph.org media allow

[uBlock-user]

you can't save a pdf from within pdfjs unless you allow Other even though you're already viewing it.

Because when you click the download button, a network request of type "Other" is fired by browser, so if other column is blocked, the download will fail. Viewing is not the same as downloading it.

The save process appears successfull.

It differs as per the browser. In Chrome, a save dialog will appear but the download will be marked Failed - blocked in chrome://downloads page.

In Firefox, the save dialog will fail with an error.

You can do one of two things either remove * 1st-party other block or add xiph.org people.xiph.org other allow

[TT1]

I concur that this does indeed occur with Firefox 67.0 after upgrading to uMatrix 1.4.0 from 1.3.16. I discovered that allowing 'other' will allow successful downloading of items with a right-click. With 'other' NOT allowed, everything appears to work, but nothing is downloaded and it shows as a fail in the download history. This does not happen in 1.3.16. I'd prefer not to have 'other' allowed at all times, not knowing what might lurk there...

[gorhill]

Firefox tells uMatrix that the network requests are of type other, it's not uMatrix who decides to label the network requests as other.

[Tiavor]

I have this problem too, but I am trying to download pictures from reddit.com and it fails with all images that are hosted on reddit. I have everything, even "other", enabled for reddit.com.
pressing the retry button in the download menu successfully downloads the image.

[TT1]

Yes, that's exactly what I need to do, click the retry. It seems to only be a problem with media files, certainly ALL image files. The problem happened simultaneous with the last uMatrix update. I'm fairly certain the issue is with uMatrix because I made no other changes other that the update.

…

On 01/12/2020 01:38 AM, Tiavor wrote: I have this problem too, but I am trying to download pictures from reddit.com and it fails with all images that are hosted on reddit. I have everything, even "other", enabled for reddit.com. pressing the retry button in the download menu successfully downloads the image. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#92?email_source=notifications&email_token=AAKVUD3KDBHXCTCX4J6WQK3Q5LQH7A5CNFSM4GMGWK52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWVWOY#issuecomment-573397819>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKVUD6TRHQNQFZGQFLJ5D3Q5LQH7ANCNFSM4GMGWK5Q>.

[Tiavor]

duh! reddit hosts the files on redd.it thus I need to whitelist that. found a few more sites where I have to whitelist them to be able to download them.

[0xpr03]

I want to give another insight, as previous reports didn't focus so much on why this is annoying:

When you try to download the certificate from here listed as direct link you get the following error. (Right-click Save as..)

uMatrix after this error (and also before)

So even if you suspect umatrix to be the culprit, you won't see anything blocked and it's really irritating that a direct interaction will fail with such an non-explaining error.

Yes the browser is probably also to blame, but the current situation can lead you to believe something completely different is broken.

[gorhill]

you won't see anything blocked

You are supposed to look at the logger, without this I have no idea that what you report is either a Firefox or uMatrix issue. The logger will show you what is being blocked and in what context.

[uBlock-user]

you won't see anything blocked and it's really irritating that a direct interaction will fail with such an non-explaining error.

Tabless/bts requests do not appear in the popup UI. So open the logger and try again to troubleshoot.

[0xpr03]

I'm sorry, yeah it's "other" again, so I guess firefox problems..

[gorhill]

It's a by-design browser behavior.

[5t3f4n]

I think I'm having the same problem but I'm not 100% sure.
It is definitely related as it's the failed file saving, and I can see the blocked "other" request in BTS. However, if I click on retry download it instantly works – without changing any rules in uMatrix.
I don't know what it does differently on 2nd try compared to 1st try.
Maybe on 2nd try it's saving from cache instead of web? But seeing as the image is already loaded it should already save from cache the 1st try instead of re-downloading I guess.

So I guess my question is this: why does it work on retry, and can I somehow automate the retry when it fails?

[0xpr03]

@5t3f4n it's a browser API Design problem which can't be circumvented except by just disabling uMatrix for a short time. The download is seen as a different source and thus blocked.

[5t3f4n]

Yes I know why it's blocked in the first place. I'm just confused as to why it works on retry download without changing anything. Notice the uMatrix icon is still there – I didn't disable it or change any rules. A way to automatically click the retry-button would effectively solve my problem.

[gorhill]

I'm just confused as to why it works on retry download without changing anything

Maybe seeing what happens in the logger would provide an answer. The logger is really the tool to use when investigating what happens in uMatrix.

[5t3f4n]

@gorhill nothing exciting in the logger.
1st try: the "other" request is blocked just like all other cases ITT. Download fails.
2nd try: nothing appears anywhere in logger in any context. Download succeeds.

I don't understand why it works on retry since "other" is still blocked with uMatrix.

[ThurahT]

This behaviour was introduced for me in "Mozilla Firefox 68.5.0esr" version. Its a browser bug.

[uBlock-user]

Click the retry icon (right hand side of downloads list).

Retry button is a part of Firefox's UI, and any requests made by Firefox's UI are intentionally not sent through the webRequest API which blocking extensions like uBO or uMatrix listen to, this is also why you can no longer block Firefox's "check for updates" network request via any extension, nor any network requests internally made by Firefox. This was possible with Firefox Legacy, but not anymore after the Quantum update with v57.