⚗[REPLAY-341] Add VisualViewport tracking (Pinch Zoom) #1118

jagracey · 2021-10-06T14:58:05Z

Motivation

Support pinch zoom and pinch scroll by tracking the "visual viewport".

As mentioned in the internal RFC "Supporting Pinch Zoom Tracking", unfortunately, the dimensions we currently record are not consistent across devices and browsers as some are referenced. On certain devices (primarily mobile), when users pinch zoom, the replay will replay incorrectly- displaying the layout viewport but not the visual viewport.

Unfortunately, browsers have not standardised various dimension properties. Mobile devices typically report dimensions in reference to the visual viewport, while desktop uses the layout viewport. For example, mobile Chrome will change innerWidth when a pinch zoom takes place, while Chrome desktop (mac) will not

With the new Viewport API, we now calculate and normalise dimension properties to the layout viewport.

Note: If the VisualViewport API is not supported by a browser, it isn't reasonably possible to detect or normalise which viewport is being measured. Therefore these exported functions will fallback to assuming that the layout viewport is being measured by the browser

Changes

Adds VisualViewport record as incremental record.
Changes innerWidth/innerHeight, scrollX/scrollY, and mouse positions to a normalised value that is referenced to the layout viewport (not pinch zoom's visual viewport) so that we do not factor in the visual viewport (pinch zoom) twice for mobile devices.

Testing

To support pinch zoom and pinch scroll unit tests, we use utilise our E2E testing framework to use Appium's gesture commands.

Unfortunately there are many pitfalls to be aware of that required work arounds:

Scrolling behaviour may disappear the address bar changing screen height
Address bar height can persist across page changes (record flushing)
FullScreen mode isn't consistent with regard to pinch zoom
Pinch/touch gestures must not exceed the bounds of the page, so are commonly repeated together.
Pinch scroll is inverted on iOS vs Android
Click() method will trigger a scroll into view, so testing the coordinate system of layout vs visual viewports for clicks doesn't appear possible
Chrome Linux doesn't support the meta viewport tag to limit max scale
Chrome Linux innerWidth vs visual.width*scale appears to differ by 15 pixels likely due to scroll bars.

I have gone over the contributing documentation.

jagracey · 2021-10-06T14:58:23Z

Reopened from #1118

codecov-commenter · 2021-10-06T15:11:53Z

Codecov Report

Merging #1118 (a91e14d) into main (514c1a2) will decrease coverage by 0.70%.
The diff coverage is 36.25%.

@@            Coverage Diff             @@
##             main    #1118      +/-   ##
==========================================
- Coverage   89.13%   88.43%   -0.71%     
==========================================
  Files          93       94       +1     
  Lines        4197     4253      +56     
  Branches      964      969       +5     
==========================================
+ Hits         3741     3761      +20     
- Misses        456      492      +36

Impacted Files	Coverage Δ
packages/rum/src/domain/record/types.ts	`100.00% <ø> (ø)`
packages/rum/src/domain/record/utils.ts	`100.00% <ø> (+16.66%)`	⬆️
packages/rum/src/types.ts	`100.00% <ø> (ø)`
packages/rum/test/utils.ts	`83.22% <0.00%> (-1.05%)`	⬇️
packages/rum/src/domain/record/observer.ts	`64.70% <23.33%> (-7.52%)`	⬇️
packages/rum/src/domain/record/record.ts	`66.66% <25.00%> (+7.14%)`	⬆️
packages/rum/src/domain/record/viewports.ts	`47.72% <47.72%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 514c1a2...a91e14d. Read the comment docs.

packages/rum/src/domain/record/observer.ts

packages/rum/src/domain/record/viewports.spec.ts

packages/rum/src/domain/record/observer.ts

packages/rum/src/domain/record/viewports.ts

init visualviewport FF

Lint fix cleanup

return data support old browsers add debug console Debug cross-browser unit test

PR review updates `window.viewport` breaks build lint

PR review fix

Bump Safari Mobile version to v13 BrowserStack device Block Safari 12 unit test: doesnt support scrollTo within iframes

init visualviewport FF

Lint fix cleanup

return data support old browsers add debug console Debug cross-browser unit test

PR review updates `window.viewport` breaks build lint

PR review fix

Bump Safari Mobile version to v13 BrowserStack device Block Safari 12 unit test: doesnt support scrollTo within iframes

jagracey · 2021-10-11T08:38:01Z

As per IRL discussions: PR drafted while we reprioritise unit testing synthetic gestures (pinch zoom + pinch scroll) via the exposed browser automation hooks in our E2E test suite.

…rowser-sdk into john.gracey/pinch-zoom-support

Debug E2E test Debug E2E test Debug E2E test Debug E2E test Debug E2E test Debug E2E test

test/e2e/scenario/recorder.scenario.ts

packages/rum/test/utils.ts

test/e2e/lib/helpers/browser.ts

test/e2e/scenario/recorder.scenario.ts

test/e2e/scenario/viewports.scenario.ts

test/e2e/scenario/recorder/viewports.scenario.ts

bcaudan

LGTM

packages/rum/src/domain/record/record.ts

BenoitZugmeyer · 2021-11-08T13:17:37Z

test/e2e/scenario/recorder/viewports.scenario.ts

+        await browserExecute(() => {
+          window.dispatchEvent(new Event('resize'))
+        })


Why is this needed?

For the record: this is to generate a ViewportResize event without having to explicitly resize the browser (which may be more complex/flaky). LGTM

test/e2e/scenario/recorder/viewports.scenario.ts

jagracey requested review from a team as code owners October 6, 2021 14:58

BenoitZugmeyer reviewed Oct 6, 2021

View reviewed changes

jagracey force-pushed the john.gracey/pinch-zoom-support branch from 5005e46 to 50786d9 Compare October 8, 2021 09:08

jagracey requested a review from BenoitZugmeyer October 8, 2021 10:23

jagracey added 7 commits October 8, 2021 12:48

Adds VisualViewport record as incremental record

a7f9412

init visualviewport FF

visual viewport normalizer implementation + test

f10520d

Lint fix cleanup

scrollX/Y fallback

54ec328

return data support old browsers add debug console Debug cross-browser unit test

Safari not handling native scrollTo

d809950

PR review updates `window.viewport` breaks build lint

Add FF to scrollX/Y()

20bd820

PR review fix

Implement cancel throttle

84ff1ff

Bump Safari Mobile version to v13 BrowserStack device Block Safari 12 unit test: doesnt support scrollTo within iframes

UA update

e3d75ac

jagracey force-pushed the john.gracey/pinch-zoom-support branch from 50786d9 to e3d75ac Compare October 8, 2021 10:48

jagracey and others added 8 commits October 8, 2021 14:12

Adds VisualViewport record as incremental record

0e48dba

init visualviewport FF

visual viewport normalizer implementation + test

ea98284

Lint fix cleanup

scrollX/Y fallback

c86966b

return data support old browsers add debug console Debug cross-browser unit test

Safari not handling native scrollTo

88ba4c4

PR review updates `window.viewport` breaks build lint

Add FF to scrollX/Y()

88094ff

PR review fix

Implement cancel throttle

1102a9d

Bump Safari Mobile version to v13 BrowserStack device Block Safari 12 unit test: doesnt support scrollTo within iframes

UA update

9c47a1f

Merge branch 'main' into john.gracey/pinch-zoom-support

60f746e

jagracey marked this pull request as draft October 11, 2021 08:36

jagracey and others added 5 commits October 11, 2021 11:05

Pinch Zoom E2E test

1277f77

Merge branch 'main' into john.gracey/pinch-zoom-support

bcc880d

Merge branch 'john.gracey/pinch-zoom-support' of github.com:DataDog/b…

7348447

…rowser-sdk into john.gracey/pinch-zoom-support

Move window.* ref into browserExec

40f560d

Debug E2E test

c4781f3

Debug E2E test Debug E2E test Debug E2E test Debug E2E test Debug E2E test Debug E2E test

jagracey marked this pull request as ready for review October 19, 2021 08:39

bcaudan requested review from a team October 20, 2021 08:01

bcaudan reviewed Oct 20, 2021

View reviewed changes

jagracey and others added 3 commits October 20, 2021 12:08

PR review fixes

610d255

Merge branch 'main' into john.gracey/pinch-zoom-support

04ff091

PR review fixes

11f2693

bcaudan reviewed Oct 21, 2021

View reviewed changes

test/e2e/scenario/recorder.scenario.ts Outdated Show resolved Hide resolved

bcaudan reviewed Oct 21, 2021

View reviewed changes

test/e2e/scenario/recorder.scenario.ts Outdated Show resolved Hide resolved

bcaudan reviewed Oct 21, 2021

View reviewed changes

test/e2e/scenario/recorder.scenario.ts Outdated Show resolved Hide resolved

bcaudan reviewed Oct 21, 2021

View reviewed changes

test/e2e/scenario/recorder.scenario.ts Outdated Show resolved Hide resolved

Split file

d6dbbc6

jagracey requested a review from bcaudan October 22, 2021 13:22

Add data prop

ace7835

bcaudan reviewed Oct 26, 2021

View reviewed changes

jagracey added 2 commits October 27, 2021 10:06

benchmark flakiness w/o sleep

cf055e8

PR review

d750494

bcaudan reviewed Nov 3, 2021

View reviewed changes

jagracey added 2 commits November 3, 2021 11:09

remove meta tests

a349dfd

renaming vars

a91e14d

jagracey requested a review from bcaudan November 5, 2021 09:30

PR review

8e079a5

bcaudan approved these changes Nov 8, 2021

View reviewed changes

BenoitZugmeyer reviewed Nov 8, 2021

View reviewed changes

jagracey and others added 4 commits November 8, 2021 14:56

PR review

12f8f5a

merge main

ae5a912

merge fix

3716aa1

Merge branch 'main' into john.gracey/pinch-zoom-support

dc63c85

jagracey merged commit 3e790f2 into main Nov 9, 2021

jagracey deleted the john.gracey/pinch-zoom-support branch November 9, 2021 08:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

⚗[REPLAY-341] Add VisualViewport tracking (Pinch Zoom) #1118

⚗[REPLAY-341] Add VisualViewport tracking (Pinch Zoom) #1118

jagracey commented Oct 6, 2021 •

edited

Loading

jagracey commented Oct 6, 2021

codecov-commenter commented Oct 6, 2021 •

edited

Loading

jagracey commented Oct 11, 2021 •

edited

Loading

bcaudan left a comment

BenoitZugmeyer Nov 8, 2021

BenoitZugmeyer Nov 8, 2021

⚗[REPLAY-341] Add VisualViewport tracking (Pinch Zoom) #1118

⚗[REPLAY-341] Add VisualViewport tracking (Pinch Zoom) #1118

Conversation

jagracey commented Oct 6, 2021 • edited Loading

Motivation

Changes

Testing

jagracey commented Oct 6, 2021

codecov-commenter commented Oct 6, 2021 • edited Loading

Codecov Report

jagracey commented Oct 11, 2021 • edited Loading

bcaudan left a comment

Choose a reason for hiding this comment

BenoitZugmeyer Nov 8, 2021

Choose a reason for hiding this comment

BenoitZugmeyer Nov 8, 2021

Choose a reason for hiding this comment

jagracey commented Oct 6, 2021 •

edited

Loading

codecov-commenter commented Oct 6, 2021 •

edited

Loading

jagracey commented Oct 11, 2021 •

edited

Loading