Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rfc][skip-ci] Screenshot Mode Service #93496

Merged
merged 16 commits into from
Apr 14, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 151 additions & 0 deletions rfcs/text/0009_screenshot_mode_service.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
- Start Date: 2020-03-02
- RFC PR: (leave this empty)
- Kibana Issue: (leave this empty)

# Summary

Currently, the applications that support screenshot reports are:
- Dashboard
- Visualize Editor
- Canvas

Kibana UI code should be aware when the page is rendering for the purpose of
Copy link
Contributor

@Dosant Dosant Mar 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if screenshot mode ownership and performance problems are essentially the same problems that we have and will have to address at some point for dashboard embed mode.

  • Loading a single dashboard in embed mode loads too much code
  • There are a lot of interactivity bugs. Most of them due to the only dashboard is aware of embed mode, but not lower-level plugins [meta] iFrame embedding inconsitencies  #93200

So I am genuinely curious if screenshot mode should be generalized into static mode. Handle all screenshots, embeds, and lighter view-only kibana use cases. Become part of the core to be able to optimize on-page load and on plugin bundling/loading level. See for example: #93496 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are similar but different requirements i think. the most notable is that in embedded mode you most likely still want to collect telemetry and probably you still want a bit more interactivity that you would in report.

capturing a screenshot. There should be a service to interact with low-level
code for providing that awareness. Reporting would interact with this service
to improve the quality of the Kibana Reporting feature for a few reasons:

- Fewer objects in the headless browser memory since interactive code doesn't run
- Fewer API requests made by the headless browser for features that don't apply in a non-interactive context

**Screenshot mode service**

The Reporting-enabled applications should use the recommended practice of
having a customized URL for Reporting. The customized URL renders without UI
features like navigation, auto-complete, and anything else that wouldn't make
sense for non-interactive pages.

However, applications are one piece of the UI code in a browser, and they have
dependencies on other UI plugins. Apps can't control plugins and other things
that Kibana loads in the browser.

This RFC proposes a Screenshot Mode Service as a low-level plugin that allows
other plugins (UI code) to make choices when the page is rendering for a screenshot.

More background on how Reporting currently works, including the lifecycle of
creating a PNG report, is here: https://github.com/elastic/kibana/issues/59396

# Motivation

The Reporting team wants all applications to support a customized URLs, such as
Canvas does with its `#/export/workpad/pdf/{workpadId}` UI route. The
customized URL is where an app can solve any rendering issue in a PDF or PNG,
without needing extra CSS to be injected into the page.

However, many low-level plugins have been added to the UI over time. These run
on every page and an application can not turn them off. Reporting performance
is negatively affected by this type of code. When the Reporting team analyzes
customer logs to figure out why a job timed out, we sometimes see requests for
the newsfeed API and telemetry API: services that aren't needed during a
reporting job.

In 7.12.0, using the customized `/export/workpad/pdf` in Canvas, the Sample
Data Flights workpad loads 163 requests. Most of thees requests don't come from
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

loads 163 requests.

And most of them performed to fetch plugin bundles?
Doesn't Headless browser support caching? puppeteer supports it
https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-pagesetcacheenabledenabled
Can we use Puppeteer or borrow their caching implementation?
If no, Core can consider serving a single bundle for all the Kiban plugins. We decided not to implement this logic when we've added long-term caching for bundle assets. So ideally, we would fix the caching problem for the headless browser.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And most of them performed to fetch plugin bundles?

That is correct. Since they are not fetched as background requests of plugins (e.g internal API requests for telemetry and newsfeed) there are limited options for a plugin to be able to reduce the 163 number.

Doesn't Headless browser support caching? puppeteer supports it
https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-pagesetcacheenabledenabled
Can we use Puppeteer or borrow their caching implementation?

Caching is disabled since there is network request interception in-place in Kibana Reporting, to implement the network policy rules, and prevent leaking credentials to 3rd parties. When network request interception is enabled, caching is disabled

Can we use Puppeteer or borrow their caching implementation?

We are using Puppeteer today.

NOTE: I just noticed this PR has been merged, which can offer help: puppeteer/puppeteer#6996

Even if we are able to restore caching, it should be understood that the cache is cleared in between Reporting jobs. The browser is "run as" multiple different users so we do not want to use a single cache for multiple jobs.

If no, Core can consider serving a single bundle for all the Kiban plugins. We decided not to implement this logic when we've added long-term caching for bundle assets. So ideally, we would fix the caching problem for the headless browser.

This would be fantastic for Reporting!

Thanks for your comments!

the app itself but from the application container code that Canvas can't turn
off.

# Detailed design

The Screenshot Mode Service is an entirely new plugin that has an API method
that returns a Boolean. The return value tells the plugin whether or not it
should render itself to optimize for non-interactivity.

The plugin is low-level as it has no dependencies of its own, so other
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note to consider:
If it is a plugin, then core can't use it and core's UI becomes an exception from the general guideline.

Header and side navigation is part of core and there is an API that apps use:

coreStart.chrome.setIsVisible(false);

I think the screenshotMode plugin shouldn't try to call this API, as there is a surface for bugs where an app's side effect makes it visible again. So probably apps should control this in the first place:

// app mount code 

if (plugins.screenshotMode.isScreenshotMode()) {
  coreStart.chrome.setIsVisible(false);
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, that's a bit of a bummer, that core's UI will have to be handled in one way (where apps call imperative core APIs), but lower-level plugins will check plugins.screenshotMode themselves.

For this particular case, it would have been a bit nicer if new screenshotMode is part of core

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apps will already control the UI, and shouldn't use this service to determine if chrome needs to be hidden.

as app is the one providing reporting with the URL, it needs to make sure that URL is printable. that means disabling the chrome. and it shouldn't use our service (well it could, but it's free to choose, most likely it will have a special url parameter or even a special endpoint for this)

maybe we should follow the same approach with other things, so app should disable telemetry in this case etc. and then we don't need the screenshotMode service at all. But this service might make it easier for us to quickly disable some plugins, but as @lukeelmers mentioned below, should be used rarely

low-level plugins can depend on it.
Copy link
Contributor

@Dosant Dosant Mar 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thought here:

We have an agreement that plugins don't manage URL and only an App should talk to the URL.
Any state propagation should be done: URL -> app -> plugin -> app -> URL. reasoning here

I am curious if this is also applicable here. and the state propagation then should be something like this (with an exception that screenshot mode plugin will read from the URL screenshot state once in setup phase):

screenshot mode plugin -> app -> all other plugins 
                            | -> core

this would also address core vs other plugins inconsistency mentioned in the comment above, without making moving screenshot mode to core.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would prefer no exception to that, the reasoning is reasonable imo :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I follow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are discussing puting a flag in the URL, and figured that would require plugin to manage the URL, but our current mentality is not to have plugins manage the URLs but have apps full responsible for it.


## Interface
A plugin would depend on `screenshotMode` in kibana.json. That provides
`screenshotMode` as a plugin object. The plugin's purpose is to know when the
page is rendered for screenshot capture, and to interact with plugins through
an API. It allows plugins to decides what to do with the screenshot mode
information.

```
interface IScreenshotModeServiceSetup {
isScreenshotMode: () => boolean;
}
```

The plugin knows the screenshot mode from request headers: this interface is
constructed from a class that refers to information sent via a custom
proprietary header:

```
interface HeaderData {
'X-Screenshot-Mode': true
}

class ScreenshotModeServiceSetup implements IScreenshotModeServiceSetup {
constructor(rawData: HeaderData) {}
public isScreenshotMode (): boolean {}
}
```

The Reporting headless browser that opens the page can inject custom headers
into the request. Teams should be able to test how their app renders when
loaded with this header. They could use a web debugging proxy, or perhaps the
new service should support a URL parameter which triggers screenshot mode to be
enabled, for easier testing.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use this param not just for testing? It would be like a real alternative to the header.

Copy link
Member Author

@tsullivan tsullivan Mar 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure there needs to be a lot of detail about how the client-side of plugin is aware that the page load request had a special header. Since there are a few options on how to do it, it comes down to internal implementation details which will get figured out during development.

I imagined there would be a server-side piece that acts like middleware to requests: runs on every request, keeps its own state, and is able to render the state into the DOM before the initial HTML is returned. The model would be like how the client-side UiSettings service works.

Looking at that now, there could be invalid assumptions about what a server-side plugin can do. I avoided proposing a URL parameter because my thinking was the URL query string is reserved for the app. If plugins are modifying it, it could create conflicts with the app. I could be wrong though.

Copy link
Contributor

@Dosant Dosant Mar 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tsullivan, is there any reason at all for it to be a header? Or could it be just a part of the URL? If it is just URL, then the whole setup seems simpler.

If plugins are modifying it, it could create conflicts with the app.

I think screenshotMode plugin could pick it up in setup and put the flag into memory. (I think this could be an exception to the agreement that plugins shouldn't mess with the URL and only apps can read/write the URL)
or
If screenshotMode is not a plugin, but part of core, then it could check and persist that query param before running any plugin, so no one would have a chance to mess it up.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we should think thru the internal implementation as well, once we agree on the public interface, which seems quite simple:

/***
* Call this method if you are a lower level plugin without an app and you want a quick way to disable yourself when kibana is in screenshot mode. Try to avoid using this, talk to app-services team when in doubt
***/
isScreenshotMode: () => boolean

so how do we get that information in there, without doing some weird hacky thing that's gonna be the source of errors and is gonna work on client and server.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ppisljar Could you please give an example of server usage?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah good point, lets not over complicate :)


# Basic example

When Kibana loads initially, there is a Newsfeed plugin in the UI that
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this particular example, would be even simpler if the newsfeed client-side plugin not loaded at all in screenshot mode?
I am curious if this is possible during "the hook-into rendering process" that you've described.
Or this could definitely be possible if screenshot thingy is part of the core and plugins can declare if they need to be skipped in screenshot mode in kibana.json

checks internally cached records to see if it must fetch the Elastic News
Service for newer items. When the Screenshot Mode Service is implemented, the
Newsfeed component has a source of information to check on whether or not it
should load in the Kibana UI. If it can avoid loading, it avoids an unnecessary
HTTP round trip, which weigh heavily on performance.

# Alternatives

- Print media query CSS
If applications UIs supported printability using `@media print`, and Kibana
Reporting uses `page.print()` to capture the PDF, it would be easy for application
developers to test, and prevent bugs showing up in the report.

However, this proposal only provides high-level customization over visual rendering, which the
application already has if it uses a customized URL for rendering the layout for screenshots. It
has a performance downside, as well: the headless browser still has to render the entire
page as a "normal" render before we can call `page.print()`. No one sees the
results of that initial render, so it is the same amount of wasted rendering cycles
during report generation that we have today.

# Adoption strategy

Using this service doesn't mean that anything needs to be replaced or thrown away. It's an add on
that any plugin or even application can use to add conditionals that previously weren't possible.
The Reporting Services team should create an example in a developer example plugin on how to build
a UI that is aware of Screenshot Mode Service. From there, the team would work on updating
whichever code that would benefit from this the most, which we know from analyzing debugging logs
of a report job. The team would work across teams to get it accepted by the owners.

# How we teach this

The Reporting Services team will continue to analyze debug logs of reporting jobs to find if there
is UI code running during a report job that could be optimized by this service. The team would
reach out to the code owners and determine if it makes sense to use this service to improve
screenshot performance of their code.

# Further examples

- Applications can also use screenshot context to customize the way they load.
An example is Toast Notifications: by default they auto-dismiss themselves
after 30 seconds or so. That makes sense when there is a human there to
notice the message, read it and remember it. But if the page is loaded for
capturing a screenshot, the toast notifications should never disappear. The
message in the toast needs to be part of the screenshot for its message to
mean anything, so it should not force the screenshot capture tool to race
against the toast timeout window.
- Avoid collection and sending of telemetry from the browser when page is
loaded for screenshot capture.
- Turn off autocomplete features and auto-refresh features that weigh on
performance for screenshot capture.