Mark McGill

Virtual Augmentations of Physical Keyboards – Less :( More 😊

The Logitech MR keyboard, used with our per-key augmentation toolkit

We’ve been doing some research into a new class of keyboard which we refer to as augmented keyboards, meaning a physical keyboard that can be tracked by a mixed reality headset, such that you can start to render virtual content on, or around, the keyboard – in effect, allowing for virtual augmentations of a peripheral. In this post, I’ll discuss the potential benefits of augmented keyboards, and go over a few key takeaway points regarding pitfalls and tips for designing per-key shortcuts, like the ones pictured below.

The benefits of Augmented Keyboards

Keyboards are great peripherals, tried and tested. They offer familiar, high-bandwidth input and are the de facto peripheral for text entry (if we squint a bit and ignore smartphones)! However, just because it works well, doesn’t mean it can’t work better.

Mixed reality headsets (e.g. VR headsets such as the Oculus Quest, or AR such as Microsoft Hololens or Magic Leap) are in their relative infancy. However, these headsets are making leaps and bounds year on year, with continual improvements in fidelity, field of view, comfort, wearability, form factor and sensing. Given this, there will inevitably come a time where our physical monitors are supplanted by these headsets for some use cases. You might arrive to work, and put on a VR headset to view your wide, virtual workspace; or you might find yourself in a coffee shop, with your AR glasses rendering a virtual display in front of you, and all you need to carry around is a keyboard instead of a laptop.

A wide VR workspace prototype we developed.

Given such headsets, we can start to fundamentally alter how we work, as we are no longer constrained to physical displays. Your desk might become interactive; virtual displays might move around you to become more accessible; changing between projects might even alter your whole workspace, with each layout customized to the needs of the project. And peripherals like the humble keyboard and mouse can now effectively operate as if they have in-built displays, meaning their very form and function can change dynamically.

One of the most fundamental changes we can imagine for the keyboard is visualizing existing, and new, shortcuts – something that has been imagined for decades, from Art Lebedev. to the latest e-ink keyboards from Nemeio. For example, the picture below shows a potential emoji layout, overloading existing key inputs. With an MR headset tracking a physical keyboard, we can render content on the keys, above the keys, or even around the keyboard, and the content doesn’t have to be 2D either. And thanks to the Logitech MR Keyboard, and before than the Logitech Bridge SDK, we can now start to prototype and design new augmented keyboard interactions such as shortcut visualization.

The Logitech MR keyboard with pass-through hand visualization, showing our emoji keyboard mapping.

This means we have a new design space to consider – how do we exploit these new capabilities to create more engaging, and more accessible, interactions with our applications? In a study looking at Microsoft Word shortcuts, we gained a few useful insights on how best to design these per-key shortcuts.

Apply Icons with care

It turns out that if we show every shortcut available, this causes a lot of visual noise and can confuse users, who are bombarded with available icons. Looking at the figure below, we can see that for common, well known shortcuts there were strong user preferences toward these not being visualized on the keyboard. Only show shortcuts that are uncommon/not well known.

Correct shortcut identification rates for a keyboard with No Icons, and with visualized Icons, from our study looking into Microsoft Word Shortcut knowledge, along with user preferences regarding how said shortcuts should be presented.

As one user pointed out:

I found that when it showed a visual aid they actually slowed me down because I was looking for the icon even though I actually knew the
shortcut.

If you provide icons, users will fall back to visual search, which is often going to be slower than relying on recall. This means that care must be taken when decided when to add Icons to the keyboard. Supporting training/recall will be a common use case. Consider Microsoft Word – actions such as copy and paste probably don’t need to be visualized for most users, and will just needlessly add to the visual clutter. However, “increase/decrease font” in the above table were relatively unknown, and could benefit substantially from being visualized, both making users aware of these shortcuts, and providing a crutch toward training users to use them.

Conversely, if we’re designing a new interface to temporarily overload the keyboard with (e.g. mapping inventory items or quick travel locations to keys on particular screens in-game) there is far less of a concern regarding the visual clutter. Users would have been performing a visual search task anyway, and these shortcuts are dynamic/infrequently used. In these cases, it might be better to keep Icons visible, creating a more interesting/engaging UI for the user.

Avoid icon ambiguity

So F5 I know is refresh, but when I saw undo I just saw the
arrow and I hit that (instead)

Visual dominance is such that ambiguous or overloaded icons can confuse users for even well known shortcuts. We took the default Microsoft Word icons for various functions such as redo, refresh, and undo, and found common conflicts there, as they are all variations on arrows. Care must be given to *what* shortcuts are visualized, and also *how* they are visualized – avoid conflicts and mis-leading/ambiguous icons. One concept we’re currently exploring here is using per-key tooltips to resolve these ambiguities (see below) but for now, visualize shortcuts clearly and sparingly.

Per-finger tracking on the Logitech MR keyboard
Using per-finger tracking to show per-key tooltips in Microsoft Word

Summary

Generally – icons are best used for infrequent actions that the user is unlikely to know, or aren’t used enough to be committed to memory yet. Don’t bombard the user unless there’s a strong underlying reason e.g. they are doing a tutorial, have explicitly requested all the shortcuts be available, or haven’t used the application in a while (e.g. reminding). If you want a pre-print draft of the full paper on this, just let me know!

Some of the keyboard mappings we’ve prototyped: secure text entry, visualizing word shortcuts, and enabling expressive emoji input.

This research was conducted by the University of Glasgow, in partnership with Logitech Design Lab, Cork.

Virtual Augmentations of Physical Keyboards – Less 🙁 More 😊

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top