xdotool is changing letters

[carnager]

teapot earnest %i earnest ~ print "Spaß" | xdotool type --clearmodifiers --file -
Spas
teapot earnest %i earnest ~ Spas

carnager@caprica ~ > printf "Spaß" | xdotool type --clearmodifiers --file -
Spa-%

Same thing without xdotool piped to another application:

print "Spaß" | cat    
Spaß

xdotool version:

xdotool -v
xdotool version 3.20150503.1

[Earnestly]

Just to add:

% xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'
àáâãäçèéêëìíîïñòóôõöùúûüÿàáâãäçèéêëìíîïñòóôõöùúûüý

% xdotool type 'øøøø'
oooo

This is likely how gconv works with the locale but I have no idea personally.

[carnager]

seems xdotool reacts differently depending on keymap used.

carnager@caprica: ~ > setxkbmap us
carnager@caprica: ~ > echo "Spaß" | xdotool type --clearmodifiers --file -
Spaß

[Earnestly]

But still fails with àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ as it outputs:

àáâãäçèéêëìíîïñòóôõöùúûüýÿàáâãäçèéêëìíîïñòóôõöùúûüý

[Moilleadoir]

I just noticed this too. It makes xdotool useless since I’m using it to type the name of a directory which of course does not exist in the lower case form.

Is there any way around this?

[fpiccinali]

use :
setxkbmap
before your call to xdotool

[Earnestly]

That doesn't help.

[fidergo-stephane-gourichon]

Result is different and surprising here on Ubuntu 16.04 AMD64, keyboard layout "fr".

printf "Spaß" | xdotool type --clearmodifiers --file -
Spqs

setxkbmap us
printf "Spaß" | xdotool type --clearmodifiers --file -
Spaß

setxkbmap fr
printf "Spaß" | xdotool type --clearmodifiers --file -
Spaß

So, setxkbmap changes something, even when setting the same keymap (i.e. supposedly no-op) since things start to work ok after it is used. Very strange.

[Anachron]

Just to add to this, even just running setxkbmap before lets xdotool type work correctly.

[Earnestly]

@Anachron No it really doesn't. This has even been suggested in this thread before, see #97 (comment)

The test case of àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ still fails after doing that.

[drzraf]

Confirming that xdotool result depends upon X keyboard layout as set by setxkbmap.
In some case, at least, it's an annoying behavior.

[fidergo-stephane-gourichon]

Indeed setxkbmap changes something

Just to add to this, even just running setxkbmap before lets xdotool type work correctly.

Indeed.

Tested again, Ubuntu 16.04 here, french keyboard layout set at X level. For the sake of completeness, running XFCE desktop environment where xfce4-keyboard-settings is set to "Use default parameters". AFAIU this means that xfce doesn't try to interfere in any way, it would just be the same with a plain X session.

printf "Spaß" | xdotool type --clearmodifiers --file -
Spqs

$ setxkbmap
$ printf "Spaß" | xdotool type --clearmodifiers --file -
Spaß

Indeed, calling setxkbmap alone is enough to change future behavior.

What does setxkbmap do?

For what it's worth, here's a glimpse at what setxkbmap does:

ltrace setxkbmap

__libc_start_main(0x400f90, 1, 0x7ffc3fe26458, 0x403180 
calloc(4, 8)                                                                    = 0x1b9e010
__strdup(0x4034e5, 0, 32, 0x7f063a3e5b20)                                       = 0x1b9e040
strcmp(".", "/usr/share/X11/xkb")                                               = -1
__strdup(0x4032be, 0x4032be, 47, 0)                                             = 0x1b9e060
XkbOpenDisplay(0, 0, 0, 0x7ffc3fe2633c)                                         = 0x1b9e080
setlocale(LC_ALL, nil)                                                          = "C"
XkbRF_GetNamesProp(0x1b9e080, 0x7ffc3fe26308, 0x7ffc3fe26310, 0)                = 1
__snprintf_chk(0x7ffc3fe25330, 4096, 1, 4096)                                   = 13
XkbRF_Load(0x7ffc3fe25330, 0x7f063a1af997, 1, 1)                                = 0
__snprintf_chk(0x7ffc3fe25330, 4096, 1, 4096)                                   = 30
XkbRF_Load(0x7ffc3fe25330, 0x7f063a1af997, 1, 1)                                = 0x1ba04b0
XkbRF_GetComponents(0x1ba04b0, 0x605380, 0x7ffc3fe25300, 0)                     = 1
XkbGetKeyboardByName(0x1b9e080, 256, 0x7ffc3fe26310, 255)                       = 0x1bc21d0
XkbRF_SetNamesProp(0x1b9e080, 0x1ba0200, 0x605380, 0)                           = 1
XCloseDisplay(0x1b9e080)                                                        = 
exit(0 
+++ exited (status 0) +++

[carnager]

sigh. No, running just setxkbmap is not enough, as @Earnestly proved above. It will fix some cases, but not all. The only way to properly fix this, is by setting your actual keyboard layout with setxkbmap.

[fidergo-stephane-gourichon]

@carnager What you write suggest (I might be wrong) that you have a clear understanding of what happens, which I clearly do not have.

If anyone can write a summary, like:

• "root cause is ...",
• "position of root cause in code is ...",
• "proper fix is ...",
• "some cases look like fixes but aren't because actually they do ..."
  ... that would be great !

PS: I'm not affected by the bug, just trying to put two cents of method. :-)

[Earnestly]

@carnager Sadly no, not even setting your actual layout will fix it. The test case I've posted multiple times seems to be ignored.

[jordansissel]

This is a long standing issue in xdotool, and I don’t know if anyone has a solution yet. Sorry. I hope we find a solution — as the maintainer I would love to see this bug resolved if we can find a way!

…

On Sun, Mar 25, 2018 at 4:09 AM Earnestly ***@***.***> wrote: @carnager <https://github.com/carnager> Sadly no, not even setting your actual layout will fix it. The test case I've posted multiple times seems to be ignored. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#97 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6ghJZRAWj2Vm6ipQ1OD8nR9xHYReks5th3rWgaJpZM4HJEJO> .

[carnager]

actually I believe this to be a XTEST bug. I tested all available alternatives back then and each had the same issue, so I don't think xdotool can do much about that.

[jordansissel]

actually I believe this to be a XTEST bug

@carnager My hunch was more around how a client (xdotool) can query xkbcommon and the older XGetKeyboardMapping.

Given users are typing correctly with their keyboard in their preferred locale, and given this problem is often resolved for xdotool by using setxkbmap, my assumption is that something is broken about the API xdotool uses for querying the keyboard map. I haven't dug much in recent years, but I did spend significant time frustrated trying to fix this problem in xdotool. I hope someone finds a solution some day :(

[Earnestly]

The problem isn't resolved by setting setxkbmap, at best it appears to be partial.

[bdantas]

@Earnestly - This problem of xdotool changing upper case Unicode characters to lower case has been a pain point for me for years, but I finally found a workaround.

First I tried xvkbd and pynput, but both of them disappointed (xvkbd sends a different character altogether, while pynput works with lower codepoint characters only).

I found xclip to be an acceptable workaround:

printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip
xdotool click 2

Running the above puts the string in X's primary clipboard (clarification here), then pastes the clipboard contents at the cursor. It results in the correct string: àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ

Limitation of this workaround is that the mouse pointer needs to be inside the window (and, if in a GUI text editor, below the line) where user is typing.

If someone has a cleaner workaround for entering Unicode characters (with correct case) into an X application via a script, please let me know.

[carnager]

@Earnestly - This problem of xdotool changing upper case Unicode characters to lower case has been a pain point for me for years, but I finally found a workaround.

First I tried xvkbd and pynput, but both of them disappointed (xvkbd sends a different character altogether, while pynput works with lower codepoint characters only).

I found xclip to be an acceptable workaround:

printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip
xdotool click 2

Running the above puts the string in X's primary clipboard (clarification here), then pastes the clipboard contents at the cursor. It results in the correct string: àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ

Limitation of this workaround is that the mouse pointer needs to be inside the window (and, if in a GUI text editor, below the line) where user is typing.

The even bigger limitation is that the application needs to support middle mouse click (think terminals with ncurses input dialog (which often needs the Shift key in addition)

[bdantas]

True, but this is the most "universal" solution I've been able to cook up with my limited knowledge of X.

If I go with the xclip approach, we have primary (selection/middle mouse click), "clipboard" (MS-style), and secondary clipboards to work with. Does one of them have a truly universal way of pasting--preferably one that does so at cursor location instead of pointer location? "Clipboard" requires Control+v in some places and Shift+Control+v in others, so no-go. Secondary clipboard seems very hard to paste from unless I'm missing something.

[carnager]

Nope, there is no universal solution. It all depends on the toolkit and implementation of the individual applications.

[bdantas]

Pity, but good to know. At least I won't chase my tail looking for a better clipboard-based workaround.

[bdantas]

I couldn't help myself and improved my workaround, anyway. All but one GUI application I tested accepted Control+v to paste from clipboard (mate-terminal is the one that didn't, but could be configured to accept it). So while this new version of the workaround may not be actually universal, it is nearly so--and the user does not have to worry about mouse pointer location because paste occurs at the cursor position:

xclip -o -selection clipboard | xclip -selection secondary # save clipboard contents
printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip -selection clipboard # put unicode characters in clipboard
xdotool key Control+v # paste from clipboard
xclip -o -selection secondary | xclip -selection clipboard # restore clipboard contents

[jordansissel]

Shift-insert is also what I use for pasting — https://github.com/jordansissel/keynav/blob/master/keynavrc#L96

…

On Fri, Mar 22, 2019 at 11:16 AM bdantas ***@***.***> wrote: I couldn't help myself and improved it, anyway. All GUI apps I tested accepted Shift+Insert to paste from clipboard. This workaround may not be any more universal, but user does not need to worry about cursor location: xclip -o -selection clipboard | xclip -selection secondary # save clipboard contents printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip -selection clipboard # put unicode characters in clipboard xdotool key Shift+Insert # paste from clipboard xclip -o -selection secondary | xclip -selection clipboard # restore clipboard contents — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#97 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6hh8hp19ObwtcSDjZOF24zb0Wwtcks5vZR37gaJpZM4HJEJO> .

[TitouanT]

Hi, I don't really know if it's the same issue but mine lead me here, and reading the discussion got me to a workaround (and maybe a solution ?).

First observation is that the example from @Earnestly will behave differently if you hold the shift key:

# don't hold shift
$ xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'
àáâãäçèéêëìíîïñòóôõöùúûüýÿàáâãäçèéêëìíîïñòóôõöùúûüý

#hold shift
$ xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'
ÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ

so it seems that xdotool print accentuated letters based on the state of the shift key.

Since I needed to make xdotool print only one letter in my script, I checked before hand the case I needed and set the shift key state accordingly:

islower=$(python -c "print('$letter'.islower())")
if [ "$islower" = "True" ]
then
	xdotool keyup Shift # because it might be down 
else
	xdotool keydown Shift
fi
xdotool type "$letter"
xdotool keyup Shift

[TitouanT]

also these four tests are interesting:

$ xdotool type 'aàÀ'
aàà # no shift
AÀÀ # shift

$ xdotool type 'AàÀ'
Aàà # no shift
Aàà # shift

$ xdotool type 'àÀa'
ààa # no shift
ÀÀA # shift

$ xdotool type 'àÀA'
ààA # no shift
ÀÀA # shift

From those, I get that capitalized ascii letter 'protect' letters to their right from the effect of the shift key.
See the next example to show that point:

$ xdotool type 'aAaàÀ'
aAaàà # no shift
AAaàà # shift

I don't know if the behavior for simple ascii characters is intended to be like that with the shift key but to get a similar behavior with accentuated characters, xdotool would only need to correctly detect the case of such characters.

[jordansissel]

Not sure if this helps, Titouan, but you can try the --clearmodifiers flag which will tell xdotool to clear any currently-held modifier keys like shift/control/alt while it types.

…

On Thu, May 21, 2020 at 10:45 AM Titouan Teyssier ***@***.***> wrote: also these four tests are interesting: $ xdotool type 'aàÀ' aàà # no shift AÀÀ # shift $ xdotool type 'AàÀ' Aàà # no shift Aàà # shift $ xdotool type 'àÀa' ààa # no shift ÀÀA # shift $ xdotool type 'àÀA' ààA # no shift ÀÀA # shift From those, I get that capitalized ascii letter 'protect' letters to their right from the effect of the shift key. See the next example to show that point: $ xdotool type 'aAaàÀ' aAaàà # no shift AAaàà # shift I don't know if the behavior for simple ascci characters is intended to be like that with the shift key but to get a similar behavior with accentuated characters, xdotool would only need to correctly detect the case of such characters. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAF2W75FJLXVGV6OET7LDRSVSCBANCNFSM4BZEIJHA> .

[TitouanT]

Thanks @jordansissel but in this case it doesn't help to output capitalized accentuated characters.

[user202729]

The bug is caused by the behavior of X:

Within each group, if the second element of the group is NoSymbol, then the group should be treated as if the second element were the same as the first element, except when the first element is an alphabetic KeySym "K" for which both lowercase and uppercase forms are defined. In that case, the group should be treated as if the first element were the lowercase form of "K" and the second element were the uppercase form of "K."

Because xdotool always find a completely-empty key code for the scratch space, the bug will always happen for single characters not already present in the keyboard mapping with "both lowercase and uppercase forms".

However, because xdotool only uses keysyms either in 0-0xff or those >= 0x01000000 (for Unicode characters), only the uppercase characters in ASCII range will cause the error.

The ranges in the patch covers all the uppercase characters in 0-0xff, so it should be good, except in the corner case that some uppercase character is already defined but in the lowercase range.

$ xmodmap -e "keycode 8 = A a"
$ xdotool type A

outputs a.

[fidergo-stephane-gourichon]

Trying to make sense of all this in layman's terms.

Experiments after JuanPotato's patch #283.

Tried current xdotool master.

TL;DR: Ran cases mentioned above, they work here after running setxkbmap (tested on XFCE and icewm sessions). Things fail when messing with the keyboard while xdotool runs.

Before setxkbmap: fail

xdotool type 'øøøø'

oooo
#Fail

echo "Spaß" | xdotool type --file -

Spqs
#Fail

sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

0áâãä972êë'úãöùúûüý
#Fail

sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

àáâãçèéêñóùúûýÿáâãäçéêìíòôö
#Fail

Running setxkbmap fixes cases in both environment.

After setxkbmap: success

echo -n "Spaß" | xdotool type --clearmodifiers --file -

Spaß
# Success

xdotool type 'øøøø'

øøøø
# Success

xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

àâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ
# Success

Messing with 'shift' key while xdotool runs: fail

# Hold shift while xdotool works
sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

0ÁÂÃÄ972ÊËÌÍÎÏÑÒÓÔÕÖ%ÚÛÜÝÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ
# Fail

# Press and release "shift" many times while xdotool works
sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

àÁÂãäçèéêëÌÍîïñòÓÔõöùúûüÝÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ
# Fail

Trying to explain simply

• Regarding lowercase/uppercase mess, I understand @user202729 explanation like this: X include some kind of "convenience shortcut" to "simplify" (avoid repeating oneself) creation of keymaps, and this caused xdotool to work in simplest cases and fail in others.

• Regarding setxkbmap I have no real clue why things work after running it. Perhaps it processes something (the in-memory keymap?) so that it meets xdotool assumptions.

Overall, I feel like anyway what xdotool does is ask X to simulate keypresses, not character inputs, in a non-atomic way. Perhaps it has no choice, that's all X protocol actually allows. Anyway this makes the whole xdotool business susceptible to race conditions between inputs from actual keyboard and simulated inputs.

More fun messing with 'Control' key

# Hold "Control" after pressing Enter, but release it as soon as you see the ^C
sleep 1 ; xdotool type 'c'

^C
$ ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

"c" continues to appear until you "type something". Also, the next actual keypress on the 'c' key would seem ignored.

I guess from X point of view, your c key just got stuck because it received a KeyDown event but xdotool was interrupted before providing the corresponding KeyUp event.

Try again and while 'c's appear click into another window where things can be types: the streams of 'c's continues there.

Conclusions

(My conclusions, anyway)

• One thing that could be needed (or not) is a kind of transactional API where X clients would send a batch of commands to be executed atomically, but I'm not even sure this would be foolproof or even desirable.
• One thing that seems better is a higher level API where you can directly inject characters or symbols, not individual keypresses, irrespective of current keymap and keyboard stats.
• X being phased out those will probably never happen.
• Writing xdotool is a tricky business. Kudos for the maintainers!

Hoping this clarifies somehow to at least one person, as the explanation by @user202729 seemed cryptic for me. Please anyone correct me if I'm wrong.