Exclusive: Designing Single-handed Shortcuts for VR & AR

Latest news

    For new computing technologies to realize their full potential they need new user interfaces. The most essential interactions in virtual spaces are grounded in direct physical manipulations like pinching and grabbing, as these are universally accessible. However, the team at Leap Motion has also investigated more exotic and exciting interface paradigms from arm HUDs and digital wearables, to deployable widgets containing buttons, sliders, and even 3D trackballs and color pickers.

    Guest Article by Barrett Fox & Martin Schubert

    Barrett is the Lead VR Interactive Engineer for Leap Motion. Through a mix of prototyping, tools and workflow building with a user driven feedback loop, Barrett has been pushing, prodding, lunging, and poking at the boundaries of computer interaction.

    Martin is Lead Virtual Reality Designer and Evangelist for Leap Motion. He has created multiple experiences such as Weightless, Geometric, and Mirrors, and is currently exploring how to make the virtual feel more tangible.

    Barrett and Martin are part of the elite Leap Motion team presenting substantive work in VR/AR UX in innovative and engaging ways.

    As we move from casual VR applications to deeper and longer sessions, design priorities naturally shift toward productivity and ergonomics. One of the most critical areas of interaction design that comes up is mode switching and shortcuts.

    Today we use keyboard shortcuts so often that it’s difficult to imagine using a computer without them. Ctrl+Z, Ctrl+C, and Ctrl+V are foundational to the efficiency of keyboard and mouse input. Most of you reading this have committed these to muscle memory.

    In VR we’ve seen controller inputs adopt this shortcut paradigm relatively easily by remapping commands to buttons, triggers, trackpads, and analog sticks. To increase or decrease the brush size in Tilt Brush you swipe right or left on the trackpad of your brush hand.

    But what happens when we think about one-handed rapid selections for bare-handed input? This requires a different kind of thinking, as we don’t have buttons or other mechanical inputs to lean on. In our previous work, we’ve mapped these kinds of commands to either world-space user interfaces (e.g. control panels) or wearable interfaces that use the palette paradigm, where one hand acts as a collection of options while the other acts as a picker.

    But if we could mode switch or modify a currently active tool with just one hand instead of two we would see gains in speed, focus, and comfort that would add up over time. We could even design an embodied and spatial shortcut system without the need to look at our hands, freeing our gaze and increasing productivity further.

    Direct Manipulation vs. Abstract Gestures

    One way to activate a shortcut with a single hand would be to define an abstract gesture as a trigger. Essentially this would be a hand pose or a movement of a hand over time. This is an exception to a general rule at Leap Motion, where we typically favor direct physical manipulation of virtual objects as an interaction paradigm over using abstract gestures. There are a few reasons for this:

    • Abstract gestures are often ambiguous. How do we define an abstract gesture like ‘swipe up’ in three-dimensional space? When and where does a swipe begin or end? How quickly must it be completed? How many fingers must be involved?
    • Less abstract interactions reduce the learning curve for users. Everyone can tap into into a lifetime of experience with directly manipulating physical objects in the real world. Trying to teach a user specific movements so they can perform commands reliably is a significant challenge.
    • Shortcuts need to be quickly and easily accessible but hard to trigger accidentally. These design goals seem at odds! Ease of accessibility means expanding the range of valid poses/movements, but this makes us more likely to trigger the shortcut unintentionally.

    To move beyond this issue, we decided that instead of using single gesture to trigger a shortcut, we would gate the action into two sequential stages.

    The First Gateway: Palm Up

    Our interaction design philosophy always looks to build on existing conventions and metaphors. One major precedent that we’ve set over time in our digital wearables explorations is that hand-mounted menus are triggered by rotating the palm to face the user.

    This works well in segmenting interactions based on which direction your hands are facing. Palms turned away from yourself and toward the rest of the scene imply interaction with the external world. Palms turned toward yourself imply interactions in the near field with internal user interfaces. Palm direction seemed like a suitable first condition, acting as a gate between normal hand movement and a user’s intention to activate a shortcut.

    The Second Gateway: Pinch

    Now that your palm is facing yourself, we looked for a second action which would be easily triggered, well defined and deliberate. A pinch checks all these boxes:

    • It’s low-effort. Just move your index finger and thumb!
    • It’s well defined. You get self-haptic feedback when your fingers make contact, and the action can be defined and represented by the tracking system as reaching a minimum distance between tracked index and thumb tips.
    • It’s deliberate. You’re not likely to absent-mindedly pinch your fingers with your palm up.

    Performing both of these actions, one after another, is both quick and easy, yet difficult to do unintentionally. This sequence seemed like a solid foundation for our single-handed shortcuts exploration. The next challenge was how we would afford the movement, or in other words, how someone would know that this is what they needed to do.

    Thinking back on the benefits of direct manipulation versus abstract gestures we wondered if we could blend the two paradigms. By using a virtual object to guide a user through the interaction, could we make them feel like they were directly manipulating something while in fact performing an action closer to an abstract gesture?

    The Powerball

    Our solution was to create an object attached to the back of your hand which acts as a visual indicator of your progress through the interaction as well as a target for pinching. If your palm faces away, the object stays locked to the back of your hand. As your palm rotates toward yourself the object animates up off your hand towards a transform offset that is above but still relative to your hand.

    Once your palm fully faces toward yourself and the object has animated to its end position, pinching the object – a direct manipulation – will trigger the shortcut. We dubbed this object the Powerball. After some experimentation, we had it animate into the pinch point (a constantly updating position defined as the midpoint between the index finger and thumb tips).

    View the original article: https://www.roadtovr.com/leap-motion-designing-single-handed-shortcuts-for-vr-ar/

    This blend of graphic affordance, pseudo-direct manipulation, gestural movement, and embodied action proved easy to learn and ripe with potential for extension. Now it was time to look at what kinds of shortcut interface systems would be ergonomic and reliably tracked from this palm-up-pinched-fingers position.

    Continued on Page 2: Spatial Interface Selection »

    The post Exclusive: Designing Single-handed Shortcuts for VR & AR appeared first on Road to VR.

    In the same category are

    This Prototype World-scale VR Game is the Size of a Football Field With recent advancements in inside-out tracking on standalone VR headsets, the tantalizing possibility of ‘world-scale’ VR experiences—tho...
    ‘FlyInside Flight Simulator’ Preview – A Promising Start in VR-native Flying FlyInside FSX, the crowdfunded VR plugin for Microsoft Flight Simulator X (2006), was built well before first-party VR motion controllers existed, dev...
    VR Support for ‘No Man’s Sky’ Being Considered by Developer No Man’s Sky, a game which has a history of calls for VR support since its 2016 release, could one day be explored in VR. Developer Hello Games ...
    Media Molecule’s ‘Dreams’ Confirmed to Support PSVR at Launch Media Molecule, the creators of LittleBigPlanet (2008) and Tearaway (2013), have been working on their upcoming game/game-creation tool Dreams for PS4...
    ‘Astro Bot’ Ranked 6th Best PS4 Game of 2018, #1 PSVR Game of All Time PSVR exclusive ASTRO BOT Rescue Mission has been making waves among critics and players alike since its release at the beginning of the month. The gam...
    Magic Leap Shows off Upcoming Enterprise Apps Focused on Professional 3D Design Magic Leap announced a number of consumer-facing apps and games coming soon to Magic Leap One, the company’s long-awaited AR headset which relea...

    Leave a comment

    Your email address will not be published. Required fields are marked *

    This site uses Akismet to reduce spam. Learn how your comment data is processed.