Gfilui

I am trying to map Caps Lock to the Down arrow key and Caps Lock with a shift modifier to the up arrow key.

I've tried using xmodmap with the following input file:

remove Lock = Caps_Lock
keycode 66 = Down Up

Pressing caps lock does work for sending the down arrow key, but pressing caps lock + shift does not send Up.

What am I doing wrong?

There will be a lot to the technical details to this solution. Feel free to jump to the layout file section if you do not care about the problem but only on the solution.

xmodmap

Moving up and down via CAPS and Shift+CAPS works on my system with the given .Xmodmap as expected.

You check whether your settings are actually in affect by calling

$ xmodmap -pk | grep ^\s*66
 66 0xffe5 (Caps_Lock) 0x0000 (NoSymbol) 0xffe5 (Caps_Lock)
$ setxkbmap -layout us && xmodmap ~/.Xmodmap
$ xmodmap -pk | grep ^\s*66
 66 0xff54 (Down) 0xff52 (Up)

the LEVEL2 Modifier

There is another (real) problem to your question that comes next:

The LEVEL2 <SHIFT> modifier which is activated by pressing the Shift button is there to visually mark characters in a text editor when moving through the lines with the arrow keys. If you assign <UP> ↑ to the second level of <CAPS>, what you are effectively doing is pressing Shift+↑. You have to deactivate the Shift modifier just for that scenario. Afaik you can not do this using xmodmap.

redirecting LEVEL2 Modifier

The "real way" of modifying key maps is with xkb. I found the solution on an old xorg mailing list. The LEVEL2 of Caps has to get redirected to the ↑(<UP>) key (The Up-Key btw has only one level.) On X.org you can deactivating the Shift modifier during redirect with the built-in function RedirectKey(key=<UP>, clearmods=Shift). clearmods "releases" the Shift modifier before the the key is actually "pressed".

layout file

The simplest way to activate the behavior is to create a new layout file:

/usr/share/X11/xkb/symbols/capslockarrow

default partial xkb_symbols "basic" 

 include "de(basic)"
 include "shift(both_capslock_cancel)"

 key <CAPS> 
 type[Group1] = "TWO_LEVEL",
 symbols[Group1] = [ Down, NoSymbol ],
 actions[Group1] = [ NoAction(), RedirectKey(key=<UP>, clearmods=Shift) ]
 ;
 ;

activate the new layout with

setxkbmap capslockarrow

Notes on the layout file

• The new layout file derives from de(basic). That is the layout for the standard german keyboard. You can add any other layout here like us(basic) or us(euro). see /usr/share/X11/xkb/symbols/ to get a glance of what is possible. The two letter code is the filename and in brackets is the respective xkb_symbols definition from the file.




• You would not have a caps key modifier anymore. Therefore I added a sort of replacement for Caps: shift(both_capslock_cancel) By pressing both LSHIFT and RSHIFT together, you can activate CAPS Lock and you release the Lock again with any other press of a Shift key.




• If anyone is wondering why I did not use a new type definition for that: I simply could not release the Shift modifier before the action of the Up Key was triggered. Releasing the Shift key though should be possible by explicitly calling preserve[Shift] = None in a spearated xkb_types.




• As stated in the comments below, Wayland will also rely on XKB but will not allow redirections.