API for dealing with 2D context loss

[SamB]

This is sort of like #2303, except not [only] for offscreen canvasses.

I wanted to add support in pdf.js to redraw pages that have suffered context loss.

Unfortunately, all I have found regarding an API for this are:

1. https://wiki.whatwg.org/wiki/Canvas_Context_Loss_and_Restoration

2. Chrome's implementation, which is behind a Runtime Enabled Feature named Canvas2dContextLostRestored.

Chrome basically has:

partial interface CanvasRenderingContext2D {
    // Context state
    // Should be merged with WebGL counterpart in CanvasRenderingContext, once no-longer experimental
    [RuntimeEnabled=Canvas2dContextLostRestored] boolean isContextLost();
}

• events contextlost (cancelable Event) and contextrestored (Event).

It would seem that context restoration is done without regard to which events have handlers registered, so long as nobody cancels contextlost. This, unfortunately, leads to confusing results when calling isContextLost from the debug console: by that point, the context has generally been "restored", whether or not the JavaScript code actually did anything. (I'm not sure what can be done about that without breaking things that already recover when they draw their next frame, just because they don't carry anything over between frames in the first place.)

Anyway, yeah. Contexts still get lost sometimes, and it'd be nice to let programs like pdf.js recover from that.

[domenic]

@whatwg/canvas, thoughts? Maybe especially @fserb since it sounds like Chrome was experimenting with this behind a flag?

[fserb]

I'm not sure what's the ask here. I think isContextLost is WAI: it represents the state where we have lost context and not restored it yet (i.e., we can't draw because the GPU went for a walk). Once our resources are back, we are back into the "ready to draw" state.

That said, we are planning to start actively working on launching context lost for 2D soon, so any feedback is appreciated.

[sushraja-msft]

Wanted to leave a short note here, I think supporting context lost on canvas is important and I would like to log my support for the API.

• In chromium whenever GPU process restarts canvas 2d loses context. This may soon be something that is programmatic since WebXR API to make webgl context xr compatible requires a GPU restart.

• Web developers are able to get into this state by allocating too many HTML canvas elements that cause the GPU process to OOM.

Work arounds like using the webgl context lost on a separate canvas to detect canvas context lost does not work (since chromium blacklists webgl support for domains after a GPU crash).

[fserb]

Working proposal for this: https://github.com/fserb/canvas2D/blob/master/spec/context-loss.md

[sushraja-msft]

Nice, looks great.

If the context was lost earlier because of the system running out of memory - would automatically "Queue a task to restore the backing buffer for context." result in triggering a new context lost ?

[RafaelCintron]

+1 to giving web developers the ability to detect whether the canvas context has been lost.

What the default behavior should be?

To not break existing content, I think the default behavior should be that the context is restored.

To keep things consistent among platforms, isContextLost should never true unless the web developer handles the contextLost event and cancels it.

If the canvas contexts will be lost for long periods of time, the spec should define what happens when web developers call canvas context functions while the canvas is lost. Same goes for using lost resources (such as patterns) with other canvas contexts that may be lost or restored. Simply ignoring calls which contain lost resources would be my preference.

[fserb]

I'm not sure if we do have a case fo "lost for long periods of time". Apart from that, I agree with what Rafael said.

[RafaelCintron]

I'm not sure if we do have a case fo "lost for long periods of time". Apart from that, I agree with what Rafael said.

By "lost for long periods of time", I mean that the developer calls cancels the contextLost event, thereby relegating the canvas to a permanent lost state.

If the context was lost earlier because of the system running out of memory - would automatically "Queue a task to restore the backing buffer for context." result in triggering a new context lost ?

If a canvas goes from alive -> lost -> alive (due to restore) -> lost , the second time it becomes lost would trigger the second contextLost event. However, if the developer ever cancels the contextLost event, the context would become permanently lost and can never be restored.

[kdashg]

If the default behavior is to auto-restore, there's not much point to a contextLost event.

What happens if the context can't be restored immediately?

[RafaelCintron]

If the default behavior is to auto-restore, there's not much point to a contextLost event.

@jdashg , for canvas contexts that developers seldom update, a contextLost event would let them know a redraw is necessary.

What happens if the context can't be restored immediately?

In practice, when will this happen? If there are, indeed, relevant scenarios, then we should split out the events into separate 'contextLost' and 'contextRestore' ones like WebGL has.

[kdashg]

A contextLost event really just says "oops, stop work, wait for the contextRestore". It's only after contextRestore that you can redraw, so the contextLost event doesn't really seem actionable (maybe you can refetch resources earlier?), unless you want to give up on the context. (context.close() would be better for this IMO)

I sort of imagine a "contextreset" that's like webglcontextrestored, with the (webgl)contextlost event auto-prevent-defaulted to incur restoration. I think that name is better too, since context-restore sounds like it recovered/restored my state, instead of resetting everything.

Firefox supports software canvas (it's the default everywhere but Windows, actually), so if the adapter/gpu takes a walk, we'll at-worst fail to allocate a gpu-backed context and give a software one.
If the adapter takes a walk but a legacy app keeps rendering, I guess we drop those calls on the floor? I guess we never standardized what happens then.
Clearly we'll need tests. :)

[RafaelCintron]

@jdashg , I'd be down with just having one event called contextReset and scuttling the cancellable behavior. This way, the event is more of a notification you can hook to know when you need to redraw everything and re-create your patterns.

For Firefox, I presume we would send the 'contextReset' event when you get a software canvas. In between the time the adapter "took a walk" and you get the software canvas (if it comes to that), I think calls should be dropped on the floor. The developer will redraw everything on contextReset anyways.

[sushraja-msft]

I like the consistency with webgl, where we have contextlost and contextrestored.

The context lost is actionable in that it informs the developer to turn off their rendering loop so as to not waste cycles drawing to a disabled context. What is the conceptual difference with 2d canvas from webgl that would warrant an inconsistency of having just a restored event for 2d canvas ?

I have a case where the web developer is using canvas elements to tile the entire working area and also creates extra caching tiles for their application. In some cases if they are too aggressive with their caching they can cause the system to run out of memory when they create caching canvases and trigger context lost. If UAs automatically restored all the contexts, it is likely that the system would run out of memory again.

I can see two ways of solving this,
a. The site gets to cancel the context lost event for some of the canvas elements that it is using for caching purposes, thereby preventing their restore and trimming the collection of caching canvases.
b. UAs restore contexts in a lazy fashion, that is even after firing context restored the back buffer is not yet allocated. Actual back buffer allocation happens when the canvas context is used post context restoration.

b. looks like additional complexity that we shouldn't impose on all UAs, therefore I think having the ability to cancel restoration (a.) is valuable for this scenario.

[kdashg]

Internally I expect all UAs to be able to handle something like (b), because without lazy allocation, every width+height resize is two allocations. I think (b) is also most backwards compatible with existing content, as existing apps that have stopped rendering to a context will stay effectively lost, but existing apps that continuously render to a context will automatically have that context restored. I see the context lost event here as having only very narrow utility. Newly written apps can use it to stop rendering while waiting for context restore, or cancel the restore, but existing (i.e. all) apps won't handle it, and we need to keep good behavior for them.

…

On Tue, Jun 9, 2020 at 9:54 AM sushraja-msft ***@***.***> wrote: I like the consistency with webgl, where we have contextlost and contextrestored. The context lost is actionable in that it informs the developer to turn off their rendering loop so as to not waste cycles drawing to a disabled context. What is the conceptual difference with 2d canvas from webgl that would warrant an inconsistency of having just a restored event for 2d canvas ? I have a case where the web developer is using canvas elements to tile the entire working area and also creates extra caching tiles for their application. In some cases if they are too aggressive with their caching they can cause the system to run out of memory when they create caching canvases and trigger context lost. If UAs automatically restored all the contexts, it is likely that the system would run out of memory again. I can see two ways of solving this, a. The site gets to cancel the context lost event for some of the canvas elements that it is using for caching purposes, thereby preventing their restore and trimming the collection of caching canvases. b. UAs restore contexts in a lazy fashion, that is even after firing context restored the back buffer is not yet allocated. Actual back buffer allocation happens when the canvas context is used post context restoration. b. looks like additional complexity that we shouldn't impose on all UAs, therefore I think having the ability to cancel restoration (a.) is valuable for this scenario. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALHJDMC2L6KC3N7N3HTBFTRVZSLZANCNFSM4IHXQWFQ> .

[grorg]

WebKit is ok with a contextRestored event also.

[grorg]

We'd need to clarify the state of the context on the restored event though. It's unlikely that it will always be able to be truly restored - it might require a full repaint, and maybe even all the properties to be reset.

In fact, if you can restore the context to a state that is indistinguishable from before the error, then I don't think there is any point even telling the page. That would leave the restored state to be a blank buffer (maybe with initial values).

[kdashg]

Yep, I would like to call the webglcontextrestored-like event (canvas)contextreset. I think that makes it more clear what happened.

[kenrussell]

When the canvas's context is lost, several requestAnimationFrame cycles might occur before the UA is able to restore it. It seems beneficial to allow the web page to potentially react to this, or cancel the auto-context-reset behavior, as @sushraja-msft pointed out above. For this reason and to retain parity with WebGL's context loss and restoration behavior, I'd like to see both 'contextLost' and 'contextReset' events specified. The WebGL spec could be updated to dispatch these preferentially instead of the existing 'webglcontextlost' and 'webglcontextrestored' events.

[kdashg]

That works for me.

…

On Tue, Jun 9, 2020 at 4:32 PM Ken Russell ***@***.***> wrote: When the canvas's context is lost, several requestAnimationFrame cycles might occur before the UA is able to restore it. It seems beneficial to allow the web page to potentially react to this, or cancel the auto-context-reset behavior, as @sushraja-msft <https://github.com/sushraja-msft> pointed out above. For this reason and to retain parity with WebGL's context loss and restoration behavior, I'd like to see both 'contextLost' and 'contextReset' events specified. The WebGL spec could be updated to dispatch these preferentially instead of the existing 'webglcontextlost' and 'webglcontextrestored' events. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALHJDNNEAENNYWFVQTXIMLRV3BATANCNFSM4IHXQWFQ> .

[RafaelCintron]

For this reason and to retain parity with WebGL's context loss and restoration behavior, I'd like to see both 'contextLost' and 'contextReset' events specified.

@kenrussell , I do not have a strong objection to having two events. However, note that the default behavior between 2D and WebGL is different today. For 2D, the default behavior is to restore the context enough that the developer can continue to render. This is why 2D content that draws to the entire canvas every frame is not broken today when the device is removed. With WebGL, on the other hand, the default behavior is that no rendering can happen when the device is removed. You have to specifically opt into contextRestored by calling preventDefault when you receive contextLost. If we just slap contextLost and contextRestored on 2D canvas with the same spec language as WebGL, we will likely break 2D sites. The best parity with WebGL we can achieve is agreeing on the names and number of the events, not the cancellable behavior.

[annevk]

@kenrussell so that means contextlost would have to fire during https://html.spec.whatwg.org/#update-the-rendering, right? If you were to queue a task for it you might end up missing frames anyway.

[kenrussell]

@RafaelCintron understood, and agreed, that the 2D canvas's default behavior has to be to restore the context, as pointed out by @sushraja-msft . It would still be a nice unification to use the same event names and number of events in both the 2D and WebGL contexts.

@annevk yes, it looks to me like both contextlost and contextreset, being events, have to fire during the "update animations and send events" step of "update the rendering". My expectation would be that contextreset would not necessarily fire immediately after contextlost, but perhaps be delayed a few frames.

[RafaelCintron]

@annevk @grorg @fserb @jdashg @kenrussell @domenic @SamB, do we agree on the following?

• 2D and WebGL canvases will support contextRestored and contextLost events.
• Spec for the events is the same as WebGL events except that the default behavior of contextLost for 2D canvas is to restore the canvas. Preventing the default means 2D canvas is not restored.
• To maintain compatibility with existing WebGL content, WebGL will continue to support webglcontextrestored and webglcontextlost.

WDYT?

[annevk]

1. Names need to be all lowercase: https://w3ctag.github.io/design-principles/#casing-rules.
2. As hinted at above the WebGL specification isn't entirely satisfactory as it doesn't integrate with "update the rendering". It should. We should also clearly define the order between these new events and the WebGL-specific events. I guess ideally we create some kind of shared algorithm that branches on the type of context as to whether to dispatch the additional WebGL-specific events.

[fserb]

@RafaelCintron what do you mean by "restore the canvas"?

[RafaelCintron]

What do you mean by "restore the canvas"?

@fserb , "restore" is not a great term because it implies the previous content remains intact, which is not true!

"Reset" is a better word to use because it implies the context has been cleared to transparent and ready to receive new rendering commands. For 2D canvas, if the web developer prevents the default for contextlost, the context ignores rendering commands or throws exceptions when in the lost state. I don't feel strongly on ignore vs. throw exception.

[fserb]

What's the use case for preventing the default?

Is there a real situation in which we can't come back up with contextRestored immediately after contextLost?

One of the arguments that was presented was to call this event contextReset, since the semantics are slightly different from WebGL.

[RafaelCintron]

What's the use case for preventing the default?
Is there a real situation in which we can't come back up with contextRestored immediately after contextLost?

@fserb , that was partially covered earlier in the issue. To summarize:

• @kenrussell and @sushraja-msft prefer that 2D canvas and WebGL to use the same events.
• @sushraja-msft is working with a web developer who caches and tiles 2D canvas elements for their working area. If the developer makes a mistake and over-caches content, it would be nice for the API not to auto-reset everything, thus getting them into the same situation which caused memory exhaustion in the first place.

I, personally, do not feel strongly about reusing the same event names, or having two events. However, I do think the auto-reset behavior should be cancellable, per the above.

FWIW, WebGPU only has lost promise, and so does not have auto-reset behavior. I think this is the best approach for a new API.

[RafaelCintron]

@fserb what are your thoughts on single vs. multiple events?

@kenrussell does the spec for the WebGL events need to be updated to satisfy @annevk 's concerns?

@jdashg and @grorg any thoughts on what is being proposed?

[kenrussell]

@RafaelCintron it's possible that the WebGL spec needs to be updated in response to @annevk 's feedback. Studying what I think is the relevant portion of the HTML spec:

https://html.spec.whatwg.org/multipage/webappapis.html#update-the-rendering

and of the WebGL spec:

https://www.khronos.org/registry/webgl/specs/latest/1.0/#5.15.2

it's not 100% clear to me what needs to be changed. The WebGL context lost and restored events use a well-defined "WebGL task source":

https://www.khronos.org/registry/webgl/specs/latest/1.0/#fire-a-webgl-context-event

@annevk if you think changes are needed could you please suggest roughly what they would be, and perhaps point to an example in another web spec?

@RafaelCintron @fserb I studied the D3D APIs and found that the similar APIs are called "device lost" and "device reset". I can understand the motivation for changing the web APIs' names, though would also suggest to use the WebGL events' precedent and call them "contextlost" and "contextrestored".

Re: only supporting context loss, not context restoration - this isn't really compatible with HTML canvas contexts, where exactly one context can be created for the lifetime of a given canvas element. WebGPU uses a different presentation model than the 2D or WebGL contexts, so is not affected by this limitation.

[kdashg]

Can we pick better names but alias them for webgl?

[annevk]

@kenrussell earlier it was suggested that these shouldn't go from a task but rather should integrate with the "update the rendering" step in the event loop processing model. The other reason changes would be needed is to have a single code path for all of these events.

[RafaelCintron]

@RafaelCintron @fserb I studied the D3D APIs and found that the similar APIs are called "device lost" and "device reset". I can understand the motivation for changing the web APIs' names, though would also suggest to use the WebGL events' precedent and call them "contextlost" and "contextrestored".

Newer D3D APIs (D3D10+) call the behavior "device removed", thus ditching the "lost" nomenclature present in D3D9 and below. Once a device becomes removed, there is no way to "reset" or "restore" it. You need to create a new one if you want to keep doing D3D things.

I do not feel strongly we should do things the way WebGL does, just because it came first. The default behavior of the events (if we have two of them) will be different and WebGPU won't even have events for this. So the web developers will already have to read the API specs carefully to know what to do.

Re: only supporting context loss, not context restoration - this isn't really compatible with HTML canvas contexts, where exactly one context can be created for the lifetime of a given canvas element.

@kenrussell , unless I am missing something, no one is suggesting we get rid of restoration. Just that we combine WebGL's two events into one event and lets you override the default behavior of restoring the 2D context.

[annevk]

@RafaelCintron I thought that in discussion it had already came up that a) you might not be able to restore for various reasons and that b) restoring might happen at a later point and that therefore having a separate event for when restoring happens would be good.

[RafaelCintron]

[@annevk said] @RafaelCintron I thought that in discussion it had already came up that a) you might not be able to restore for various reasons and that b) restoring might happen at a later point and that therefore having a separate event for when restoring happens would be good.

@annevk , earlier in the thread, @fserb asked "Is there a real situation in which we can't come back up with contextRestored immediately after contextLost?" His question was never answered. I, too, would like to know the answer to his question.

If it is true that we may not able to restore the canvas either at all, or with a substantial delay, then I agree we should have two events.

FWIW, in legacy Edge, we were always able to reset 2D and WebGL canvas with either a hardware or software implementation. Developers were never left with no canvas context.

[kdashg]

OpenGL specs leave wiggle-room about allowing a delay between loss and restorability, so I'm hesitant to require immediate restoration that might incur a hardware ->software downgrade if the hardware is allowed to take time to come back online.

[kenrussell]

In Chromium there are absolutely situations where (in WebGL, at least) webglcontextlost and webglcontextrestored can't be delivered atomically - for example, when the GPU process takes some time to re-launch. It might be the case that several requestAnimationFrame cycles elapse between them, or it might be the case that webglcontextlost is delivered and the user agent decides not to allow restoration. Chromium's GPU-accelerated 2D canvas implementation has similar characteristics, so it seems impractical to coalesce the two events. I think they should remain separate.

@annevk I'd greatly appreciate a pointer into the HTML spec showing one event which integrates properly with the "update the rendering step" in the event loop processing model. I don't understand yet what changes need to be made to the WebGL spec. Alternatively, if you can suggest changes to:

https://www.khronos.org/registry/webgl/specs/latest/1.0/#5.15.2
https://www.khronos.org/registry/webgl/specs/latest/1.0/#fire-a-webgl-context-event

I'll be happy to make them.

[annevk]

See how https://html.spec.whatwg.org/#update-the-rendering invokes various algorithms to run them at "layout time", such as animation frame callbacks, fullscreen, etc. A decision should be made relative to those events when canvas things get to run and then the specification can be updated to that effect. I.e., add another call for "canvas things" and define elsewhere what "canvas things" means.