click function: strange meta keys in log and does not always click

[romangrothausmann]

When using SikuliX (1.1.4) inside a docker image (https://gitlab.com/romangrothausmann/dind-sikuli) for automated fog testing in GitLab CI with Xvfb, I noticed that the click function does not always lead to a click event on the running GUI even though the mouse pointer moved to the correct click position.
I found some similar reports but only RaiMan/SikuliX-2014#260 seems related. By repeating the GL CI job a few times (3-6), some succeed all right. I observe this problem also on a slower laptop running sikuli on a default X-server (even if running the sikuli script in "slow motion").
It noticed that often (possibly always) the logs report some meta keys being held while clicking (even though that is not part of the sikuli script), e.g.:

[log] CLICK on L(955,281)@S(0)[0,0 1280x1024] (591 msec)
[log] Meta+Alt Graph+CLICK on L(300,953)@S(0)[0,0 1280x1024] (728 msec)
[log]  TYPE "Input.txt"
[log] CLICK on L(250,972)@S(0)[0,0 1280x1024] (570 msec)
[log] Meta+Alt Graph+CLICK on L(300,953)@S(0)[0,0 1280x1024] (601 msec)

However, I do get the very same log messages even for jobs that were executed correctly and succeeded.

Any ideas what could be causing this click effect variation?

[romangrothausmann]

Also tested on a fast PC with default X-server, same problem, so the speed seems not to be related to this issue.
I just found a slight hint to a possible cause for this. Here https://bugs.launchpad.net/sikuli/+bug/1025991/comments/6 it seems from what is reported that removing the timeout given to the click function removed the meta keys from being held. While this seems to reduce the likelyness of meta keys being held (according to the logs), I do observe the issue (that no click event reaches the app) also for click() without a timeout specified. What does help is to follow with a click(Mouse.at()), e.g.

click(Pattern("clickpos.png").similar(0.95).targetOffset(0, -10), 100) # text field needs click to get focus
click(Mouse.at()) # just to make sure that text field got a click

[RaiMan]

What do you intend by the 100 as second parameter of the click()?
The parameter is taken as specifying the modifier key(s) to be pressed while clicking.
It should be one of the KeyModifier.SHIFT, .ALT, .CTRL, .META (Mac: CMD, Win: WIN) or a sum of more of them.

So your 100 leads to the press-and-hold of META (ignored on Linux currently. Should hold CTRL I think - so a bug?) and the ALT-GRAPH key (right ALT key I think).
I do not think this is your intention ;-)

So IMHO this should work:

click(Pattern("clickpos.png").similar(0.95).targetOffset(0, -10))

BTW: your above workaround should have been

hover(Pattern("clickpos.png").similar(0.95).targetOffset(0, -10), 100) # only move the mouse
click(Mouse.at())  

... which of course would have given you an error about the 2nd parameter ,100 ;-)

[RaiMan]

... and I admit, that the logging in this case should be improved ;-)

[romangrothausmann]

Many thanks @RaiMan for your reply and pointing out that the click function does not take a timeout as second parameter but instead modifies. Not sure where I got it from because it is documented all correctly but it seems the very same mistake was made here: https://bugs.launchpad.net/sikuli/+bug/1025991/comments/6. Probably a remainder from a former wait() or exists() and the general python disadvantage that it does not check for types in function calls.

[RaiMan]

Hope it works now.

I will revise the handling of the second click parameter a bit, so it creates an error message if the given value does not make any sense (value can max be 15 if you want to hold all 4 possible keys ).

BTW: was the intention of the 100, that SikuliX should wait max 100 secs for the image to appear?

[romangrothausmann]

BTW: was the intention of the 100, that SikuliX should wait max 100 secs for the image to appear?

Yes, basically setting the default timeout but just for this call.

Hope it works now.

Sadly not, the actual problem that the running app does sometimes not receive a click event still persists even now where no meta keys are held during click. Only a directly following click(Mouse.at()) works.
Should I re-report that on a new issue since the issue title has changed?

[RaiMan]

app does sometimes not receive a click

There are possible reasons:

• GUI not (yet) ready (the click is ignored)
• app does not have focus (the click only brings the window to foreground)

Since SikuliX currently does not have a feature to "know", wether a click really happened against the intended app/GUI, it is very hard, to debug the reason behind.

So if it is really a problem for you, you should create your own click function, that divides the click into its possible steps and allows some intermediate waits and logs.

• search an image using wait(image, timeout)
• if found:
  • hover (move mouse) to the target
  • wait a short time if needed (up to 0.5 secs)
  • click(Mouse.at())
  • wait a short time if needed
  • check for expected GUI changes (meaning click worked)

Should I re-report that on a new issue

If it only happens sometimes, but sometimes works, it surely is not a SikuliX problem.
So no extra issue.

But feel free, to post it on Launchpad as a question, so others have a chance, to talk about their experiences. Generally it is very hard to really help in such situations, since only you know/see what happens.

If you finally think it is a SikuliX problem: feel free to report it here or on Launchpad with as much information as possible about environment, app context and script snippets.

[romangrothausmann]

Many thanks @RaiMan for looking into this. I will test how far I can get with a specialized click() function. I have the hope that waiting between hover and click(Mouse.at()) might help.
Is there already a delay in the standard click() function between reaching the target position and issuing the click event?

[RaiMan]

Is there already a delay in the standard ...

No, not an intended wait, only some milliseconds caused by the internal handling (Java AWT Robot).

[xywang68]

I can certainly confirm that this issue also exists in robotjs independent form sikulix. I can reproduce the same scenario in both applications.
However, with the earlier sikulix jar somehow it avoided the robotjs issue -- at that time I can reproduce this issue only with robotjs. With sikulix the same scenario magically worked.
But now it breaks in both.

The test scenario involves clicking the image of a drop down list. In my case is the Selenium console "Create a New Session" modal browser drop-down. See the picture in this link:
https://raw.githubusercontent.com/Yash-777/SeleniumDriverAutomation/master/images/wiki/NON_GRID.png

A few weeks ago using sikulix I am able to see the drop-down list. Now I cannot, because the click to display the drop-down does not register by the browser. It shows a red shadow as if the drop-down will display.

The same scenario has been failing with robotjs since the beginning. As I am digging more I will file this bug to robotjs.

I am not 100% sure if the change of behavior on sikulix side is due to change is sikulix jar file or due to I updated the npm and node environment. I will try to restore to the same npm/node and try out again.

Will keep you updated here.

[xywang68]

Hey, I have confirmed that roll back java, npm and node versions doesn’t not help, so the only last resort is to get an older version of the 1.1.4 jar file.
@RaiMan, can you provide me a link to download a 1.1.4 jar file that was built around December time frame?

[xwmbww]

MP4 for passed on DISPLAY=:0
https://ufile.io/4ddoo

MP4 for failed on xvfb-run --server-num=99
https://ufile.io/24ny1

MP4 files can be played by chrome browser directly.

[xwmbww]

btw I have file a ticket to robotjs
octalmage/robotjs#453

[xwmbww]

Correction:
This issue does not exist at first when we run xvfb-run in single run. It occurs when we tried to execute xvfb-run command in background and in parallel (multiple xvfb-run). After that, the issue persist regardless single or parallel xvfb-run.