Interaction with head-mounted displays (HMDs) places a significant workload on neck muscles, exacerbated by the extra weight of the HMD. This paper introduces NeckErgonomics, an interaction approach that adapts HMD interfaces according to estimated neck muscle electromyography (EMG) signals to improve neck ergonomics. Leveraging an open-source dataset, we train a lightweight neural regression model to predict relative neck muscle activity from head yaw, head pitch, and fixation duration. We demonstrate NeckErgonomics through three model-informed applications: (1) control–display gain for head-based pointing, (2) dynamic recentering of virtual windows during knowledge work, and (3) adaptive viewport amplification for seated 360° video viewing. We evaluated the applications in a controlled within-subjects user study (N=18) to demonstrate the viability of NeckErgonomics. Across the three tasks, NeckErgonomics led to lower accumulated head movement, sustained head rotation, and perceived physical demand. The results also reveal task-specific trade-offs with speed, precision, visual stability, and sickness. Taken together, this work indicates a path toward ergonomics-by-design XR techniques in which adaptation is grounded in the estimated muscle activity rather than ambiguous behavioural cues.

Guanlin X. Li, Florian Weidner, Anam Ahmad Khan, Haopeng Wang, Jinghui Hu, and Hans Gellersen. 2026. NeckErgonomics: Adapting Head Rotation in HMDs with Predicted Neck Muscle Activity. In Proceedings of the 2026 ACM Symposium on Spatial User Interaction (SUI '26), Bari, Italy. Association for Computing Machinery, New York, NY, USA, Article 24, 11 pages.

Gaze has been integrated to assist 3D object manipulation with the hand for fast and effortless selection and radial movement, yet depth translation still largely depends on hand input. In this work, we investigate using gaze for radial redirection and head pitch for depth control together to complement 6DoF hand manipulation by introducing MagicPitch – a trimodal technique that integrates gaze, pinch, and head pitch. We conducted a 3D docking task (N=24) to evaluate MagicPitch alongside its variant, which scales the pitch-depth effect via hand movement to improve stability. Our results show that redistributing depth control to head pitch significantly reduces hand effort with a trade-off to slightly increased neck motion, highlighting the potential of head–hand cooperation for fatigue-aware multimodal XR interaction. Overall, the findings demonstrate the effectiveness of trimodal integration of gaze, hand, and head pitch input for 3D object manipulation and suggest a general principle of complementing 6DoF hand input with gaze and head input to offload manual effort.

Haopeng Wang, Pavel Manakhov, Thorbjørn Mikkelsen, Florian Weidner, Ken Pfeuffer, and Hans Gellersen. 2026. MagicPitch: 3D Object Manipulation with Gaze, Pinch, and Head Pitch. In 2026 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE. To appear.

Traditional VR viewport control primarily relies on head and torso movement, which can be effortful and limiting in both constrained and extended-use settings. We introduce Looking Around by Looking Around (LALA), a gaze-based VR pitch-and-yaw viewport control technique designed for natural and effortless omnidirectional exploration via eye movements, without requiring or obstructing movement of the head, hand, or body, offering a low-effort and highly accessible interaction method. Because gaze is primarily used for perception and exhibits oculomotor and perceptual asymmetries, using it directly for control is difficult. To address this, we designed an asymmetric omnidirectional control profile for the eye, then built on it to exploit tendencies for eyes to stay within comfortable regions for viewport control. We evaluated LALA in a user study (N = 18) featuring two contrasting tasks: alignment towards known directions and open-ended visual search towards unknown directions. LALA was strongly preferred over the traditional baseline, achieving competitive performance while enabling fully hands-free interaction with minimal physical movement.

Hock Siang Lee, Jinghui Hu, Florian Weidner, Haopeng Wang, and Hans Gellersen. 2026. Looking Around by Looking Around: Omnidirectional Gaze-based VR Viewport Control. IEEE Transactions on Visualization and Computer Graphics (early access), 1–11.

AR Widgets placement strategies for AR glasses. A) View-Fixed placement: AR Widgets are displayed as a head-anchored overlay in the upper part of the user’s field of view (FoV). B) Look-Down placement: AR Widgets are placed on the ground in front of the user when looking downwards. They are fixed in world coordinates but repositioned once the user passes them. C) Wall-Aligned placement: AR Widgets are attached and aligned to the physical wall in front of the user. Similar to the Look-Down strategy, the widgets are fixed to the physical world and repositioned when leaving the user’s FoV. D) View as captured through the AR glasses using the Look-Down placement.

Future lightweight smartglasses will enable a continuous augmentation of the user’s view with digital information. However, the effective integration of persistent overlays in Augmented Reality (AR) remains insufficiently understood. Where should frequently accessed persistent information, like widgets and information panels that do not refer directly to objects in the environment, be displayed in the user’s view? In this work, we explore placement strategies for "AR Widgets" that, similar to traditional information widgets, should be easy to access. We propose two novel environment-aware placement strategies that dynamically reposition world-referenced widgets, and compare them to a more traditional view-fixed placement method with a user study (n=18) across sitting and walking usage scenarios. Our findings indicate that the choice of widget placement strategy impacts user experience, distraction, placement, and clear-view ratings, with the wall-aligned method scoring the highest.

Stefanie Zollmann, Tobias Langlotz, Daniel Wagner, Pavel Manakhov, and Wei Hong Lo. 2026. AR Widgets: Exploring Placement Strategies for Persistent Widgets for Augmented Reality. In Proceedings of the 28th International Conference on Mobile Human-Computer Interaction (MobileHCI '26). ACM, New York, NY, USA, Article MHCI9650, 22 pages.

Core challenges for hands-free techniques: (a) Disambiguating Select vs. Drag where UI context is insufficient, e.g., an app icon can be selected to launch or dragged to reposition. (b) Supporting UI-independent, free-form tasks, including precise dragging (time-seeking) and lasso selection (circle-and-search). Cluttered controls further highlight the need for precision.

Hands-free interaction is essential when users’ hands are occupied with primary tasks. A key challenge is unifying precise, UI-independent selection and continuous dragging without explicit mode switching. This paper introduces two novel techniques: Aligner (spatial eye-head alignment) and Nodder (gestural decomposition of a nod); and evaluates them against an established Winker technique, a gestural clutch using single-eye closure. A user study with controlled Fitts’ law and dragging tasks, alongside a practical application, evaluated the techniques. Results show that Aligner offers superior selection precision but creates a visuomotor conflict during dragging; Nodder optimizes movement efficiency for large-amplitude targets despite higher physical effort; and Winker provides the fastest performance but is susceptible to accidental activation. These findings inform the design of future hands-free systems by highlighting the context-dependent nature of this interaction challenge.

Baosheng James Hou, Pavel Manakhov, Jinghui Hu, Haopeng Wang, and Hans Gellersen. 2026. Aligner, Nodder, and Winker: Creating Complete, Hands-Free Interaction Techniques that Unify Precise, UI-Independent Selection and Dragging. Proceedings of the ACM on Human-Computer Interaction 10, 3, Article ETRA016 (2026), 21 pages.

A gaze attention shift (e.g., from a blue square to a red circle) can be the result of a user’s planned action, or an involuntary response to the visual stimulus. We propose classifying shifts as user-directed or object-guided within a pipeline.

Gaze can be directed at will, or guided by objects that draw our attention. We propose classifying gaze shifts as either directed or guided, since these two forms of attention have different implications for HCI. Directed attention may serve as stronger indicator of users’ planning and intent, whereas guided attention reflects interface efficacy in guiding information acquisition. We introduce a method based on eye and head movement features during gaze shifts, using data collected in virtual reality to train and evaluate a machine learning model, which we then validate in application to visual search. Our results show that this classification is both feasible and practical, extending established uses of eye tracking in HCI. This is significant because it enables a new level of analysis of visual attention.

Anam Ahmad Khan, Baosheng James Hou, and Hans Gellersen. 2026. Directed or Guided? Classification of Gaze Attention Shifts based on Eye and Head Movement. Proceedings of the ACM on Human-Computer Interaction 10, 3, Article ETRA002 (2026), 19 pages.

Visual alignment tasks, such as perspective pointing, inherently involve ambiguity because either the left or right eye can dominate, serving as the vantage point. Previous work presented eye dominance as dynamic, influenced by horizontal gaze angle to the target. In this work, we investigate prediction based on both target-based and user-dependent features, with machine learning models trained from data collected from a perspective-pointing task (N=28). We confirm that vertical target angle influences eye dominance, strengthening the effect of horizontal angles, specifically below eye level. Models using target information alone performed poorly, particularly for left-eye cases. Including user-specific features improved accuracy and class balance, with gradient boosted classifiers achieving the highest performance. These results underscore the significance of personalized features for eye dominance prediction, with practical implications for virtual reality (VR) interfaces.

Franziska Prummer, Jinghui Hu, Florian Weidner, and Hans Gellersen. 2026. Prediction of Eye Dominance in VR. In Proceedings of the 2026 Symposium on Eye Tracking Research and Applications (ETRA '26), Marrakesh, Morocco. Association for Computing Machinery, New York, NY, USA, Article 89, 10 pages.

Multimodal interaction has long promised to make interfaces more intuitive and effective by combining complementary inputs. Among these, gaze and speech form a compelling pairing: gaze provides rapid spatial grounding, while speech conveys rich semantic information. Together, they offer rich cues for understanding user behaviour and intent. Yet despite decades of exploration, the research remains fragmented, making this synthesis timely as these inputs mature and are integrated into consumer-ready devices. This scoping review examined 103 studies published between 1991 and 2025, organised into explicit, where users intentionally provide gaze and speech, and implicit, where systems leverage users’ natural behaviours to support interaction. Across both, we identified recurring ways for combining gaze and speech to resolve ambiguity, ground references, and support adaptivity. We contribute a synthesis of research on their combined use while highlighting challenges of temporal alignment, fusion and privacy, offering guidance for future research toward richer multimodal human-computer interaction.

Anam Ahmad Khan, Florian Weidner, Jungwoo Rhee, Yasmeen Abdrabou, Andrea Bianchi, Eduardo Velloso, Hans Gellersen, and Joshua Newn. 2026. Gaze and Speech in Multimodal Human-Computer Interaction: A Scoping Review. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, Spain. Association for Computing Machinery, New York, NY, USA, Article 672, 24 pages.

Extended Reality (XR) headsets enable large, reconfigurable multi-display workspaces and support view manipulation, allowing the workspace to reposition itself around the user. Cursor warping similarly reduces traversal distance and pointer search by reinitialising the cursor at defined locations. Yet when both mechanisms operate together, the spatial relationship between user, displays, and cursor becomes dynamic, and it remains unclear how cursor repositioning behaves when the workspace itself moves. In a study (N=20) of five cursor-warping strategies with two view manipulations, we show that the benefits of both do not automatically combine: workspace motion can disrupt spatial consistency and alter both performance and movement costs. We show that continuous cursor movement in world space is limited compared to alternative warping techniques, and cursor behaviour and view control are tightly coupled. Hence, cursor initialisation and view manipulation must be co-designed to support efficient and comfortable interaction in XR multi-display environments.

Yuzheng Chen, Haopeng Wang, Hock Siang Lee, Florian Weidner, Jinghui Hu, and Hans Gellersen. 2026. The Interaction of Cursor Warping and View Manipulation in Multi-Display XR Environments. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, Spain. Association for Computing Machinery, New York, NY, USA, Article 458, 17 pages.

People differ in how much they move their head versus their eyes when shifting gaze, yet such tendencies remain largely unexplored in HCI. We introduce head movement tendencies as a fundamental dimension of individual difference in VR and provide a quantitative account of their population-level distribution. Using a 360° video free-viewing dataset (N = 87), we model head contributions to gaze shifts with a hinge-based parametric function, revealing a spectrum of strategies from eye-movers to head-movers. We then conduct a user study (N = 28) combining 360° video viewing with a short controlled task using gaze targets. While parameter values differ across tasks, individuals show partial alignment in their relative positions within the population, indicating that tendencies are meaningful but shaped by context. Our findings establish head movement tendencies as an important concept for VR and highlight implications for adaptive systems such as foveated rendering, viewport alignment, and multi-user experience design.

Jinghui Hu, Ludwig Sidenmark, Hock Siang Lee, and Hans Gellersen. 2026. The Eye–Head Mover Spectrum: Modelling Individual and Population Head Movement Tendencies in Virtual Reality. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, Spain. Association for Computing Machinery, New York, NY, USA, Article 613, 18 pages.

With a stylus, users can both sweep sketches across models and pinpoint locations with precision. Building on this dual capability, we explore how teleportation can be integrated into stylus interaction without disrupting the flow of common stylus usage. We introduce two key ideas: flipping the stylus as an intuitive mode switch between drawing and teleportation, and using gaze to set orientation while the stylus handles positioning. In a user study that features a teleport-and-orient task, we evaluate six teleportation techniques, covering two mode-switching methods (Button and Flip) and three orientation approaches (StylusRoll, StylusPoint, and GazePoint). The results offer new insights into the relative merits and limitations of each technique. Our work contributes to knowledge about teleportation in VR and fills the gap in seamlessly integrating teleportation with stylus use in 3D.

Yang Liu, Qiushi Zhou, Mathias N. Lystbæk, Aidan Kehoe, Mario Gutierrez, Hans Gellersen, and Ken Pfeuffer. 2026. StylusPort: Investigating Teleportation using Stylus in VR. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, Spain. Association for Computing Machinery, New York, NY, USA, Article 1678, 12 pages.

Effortless manipulation informs and relies on preshaping: the subconscious posing of the hand before grasping. Virtual environments and the design of interaction techniques alters interaction requirements like contact and reach, forcing behavioural adaptation. We present a comparative study investigating preshaping behaviour across direct versus indirect (gaze-assisted) and bare-hand versus controller techniques on a docking task. Results reveal that response patterns scale with anticipated task difficulty, and that direct techniques elicit effective posing of the hand. Indirect techniques shortcut hand transport and, in turn lacks the sensory feedback to guide planning, inducing efficient but attenuated responses that necessitate compensatory manipulation and clutching. Notably, controllers that afford in-hand rotation allow users to extend their range of motion. These findings can inform interaction design to better afford preshaping and optimise 3D manipulation tasks.

Thorbjørn Mikkelsen, Qiushi Zhou, Mar Gonzalez-Franco, Hans Gellersen, and Ken Pfeuffer. 2026. Preshaping Hand Behaviour for Direct and Indirect Manipulation of 3D Objects. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, Spain. Association for Computing Machinery, New York, NY, USA, Article 1692, 13 pages.

An example of Controller Drag, a technique which alters the viewport rotation by coupling it to the pose of an external controller. (Left) The user points at a location in VR and enters "Dragging Mode", often by holding down a button on an external controller. (Right) While in "Dragging Mode", the user "drags" the world right by moving the controller, effectively rotating the viewport to the left by an equivalent amount. To achieve larger cumulative rotations, users would often clutch by releasing the controller input, moving their hand back to a comfortable position, and then re-engaging the input.

Looking around is fundamental to VR experiences, yet head rotation alone can be limiting due to comfort, posture, and accessibility constraints. Although numerous alternative viewport control techniques have been proposed in academia, it remains unclear which are actually implemented in consumer-facing systems. We present a systematic survey of viewport control techniques in 28 VR 360° video players available on major platforms. We identified and classified implemented techniques, revealing that only half of surveyed players supported alternatives to 1:1 head rotation by default, and all such alternatives were controller-based. Among these, Controller Drag is the most common, whereas Controller Snap—prominent in academia—appears less frequently in practice, highlighting the gap between research and real-world deployment while raising concerns about the limited diversity of techniques in current consumer-facing systems. Our findings provide an overview of current industry practice and inform the design of future viewport control techniques.

Hock Siang Lee and Hans Gellersen. 2026. How do we Look Around? A Survey of Viewport Control Techniques in 360° Video Players. In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Daegu, Republic of Korea. IEEE, 99–105.

Head pointing is widely used in VR, but the standard 1:1 headcursor mapping can result in large head movements, which may increase discomfort due to HMD weight and shifted balance. To address this, we introduce StrainGain, an ergonomics-first concept that adapts control-display (CD) gain according to estimated neck strain. We operationalise StrainGain with a lightweight regression model trained on an open-source neck EMG dataset. The predicted strain drives a cursor amplification. We evaluated the technique in a 2D Fitts' law-like head-pointing task against 1:1 mapping and HeadShift dynamic gain (N=18). Results show that strainaware amplification significantly reduces accumulated head movement relative to both baselines, with the strongest reduction at the largest amplitude (100°), accompanied by acceptable performance trade-offs. Overall, StrainGain demonstrates how physiological-state-informed gain control can reduce head movement during VR pointing, reframing gain adaptation as a mechanism for sustained comfort and well-being rather than performance alone.

Guanlin Li, Florian Weidner, Hans Gellersen, Haopeng Wang, Anam Ahmad Khan, and Jinghui Hu. 2026. StrainGain: Neck Strain Amplifies Head Pointing in VR. In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Daegu, Republic of Korea. IEEE, 362–367.

Ground pointing in VR typically relies on raycasting or parabolic pointing with absolute mappings. While relative mappings have been effective in other pointing and depth-control tasks, their role in ground pointing remains underexplored. We investigate speed-sensitive relative mappings for parabolic ground pointing and implement three variants: Pitch (controller pitch angle), PAD (trackpad-based depth control), and Pad&PITCH (combining both). In a study with target distances of up to 9 m, relative mappings reduced hand effort but compromised speed and accuracy. We suggest that perceived sensitivity scaling, offset accumulation, and controller height should be integrated into future designs.

Haopeng Wang, Alexandra Gerbert, Florian Weidner, Yasmeen Abdrabou, Anam Ahmad Khan, Hock Siang Lee, and Hans Gellersen. 2026. Exploring Relative Mapping for Parabolic Ground Pointing in VR: Lessons Learned and Future Directions. In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Daegu, Republic of Korea. IEEE, 1065–1066.

Introduction: Head-mounted displays (HMDs) are central to virtual and augmented reality (VR/AR) experiences, yet their reliance on head movement for interaction imposes physical demands on the neck. While neck pain is already a prevalent musculoskeletal issue, neck muscle activity during HMD usage remains poorly understood. Methods: In this paper, we present the first empirical study of neck muscle activity during head fixation in a seated VR pointing task. Using surface electromyography, we measured muscle activity across 67 head poses in 15 participants. Results: Our findings describe how neck muscle workload increases with rotation angle, is asymmetric across directions, and exhibits distinct temporal dynamics depending on head rotation angle. We contribute an open dataset and a detailed characterisation of neck muscle activity during prolonged head fixation, providing a physiological baseline for future research on the ergonomics of immersive technologies. Discussion: Our findings highlight how specific head orientations and fixation durations differentially load the neck musculature, offering the basis for a physiological explanation for discomfort during prolonged VR use and establishing a baseline for future ergonomic assessments.

Guanlin Li, Florian Weidner, Jinghui Hu, and Hans Gellersen. 2025. Quantifying neck muscle activity during head fixation in VR. Frontiers in Virtual Reality 6, 1682866.

Gaze is fast and intuitive for raycasting, but lacks a way to control depth for input in 3D. We propose HeadDepth to augment gaze with vertical head rotation (pitch) to control depth along the line of sight, and investigate three pitch-to-depth mappings: Relative maps pitch velocity to changes in depth; EdgeGain adds dynamic gain dependent at eccentric gaze angles; and PingPong provides an absolute back-and-forth mapping that repeats at different pitch angles. HeadDepth is not as fast as controller-based RayCursor but has the advantage of being hands-free. In a user study, all three variants proved effective for 3D positioning; PingPong required least effort and EdgeGain was least affected by differences in tasks. Eye-head coordination and task had a significant effect, as head movements in support of gaze can affect depth control synergistically or antagonistically. Our results have significance beyond HeadDepth as they generalize to any interaction where head rotational input concurs with gaze fixation on visual feedback. Interaction designs may benefit from dynamic adjustment to eye-in-head angle to prevent discomfort and maintain objects within the user's field of view.

Haopeng Wang, Florian Weidner, Yasmeen Abdrabou, Ken Pfeuffer, and Hans Gellersen. 2025. HeadDepth: Gaze Raycasting with Head Pitch for Depth Control. In 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Daejeon, Republic of Korea. IEEE, 370–380.

To efficiently manipulate in 3D space, current XR systems support three common input methods for 6 degrees-of-freedom (DOF) object manipulations: direct grasping with a unified input/output mapping, as well as gaze and hand-ray-based input where the hand is decoupled from the manipulation object. In our study, we examine how this affects 3D object docking (translation + rotation) tasks, for both DOF integration and separation mappings to uncover the speed-accuracy performance trade-offs.

Hand-tracking based 3D object manipulation in Extended Reality (XR) typically employs a pinch gesture for acquisition and manipulation through a direct mapping from 6-degrees-of-freedom (DOF) hand movement to that of the object. In this work, we investigate the effect of separating this mapping to concurrent 3DOF controls (DOF-Separation) of translation and rotation of the virtual object using the position and orientation of the hand independently. We aim to understand how DOF-Separation could ease manipulation for different techniques with varying requirements for hand position and orientation during acquisition, including Virtual Hand, Hand Ray, and Gaze&Pinch. Through a user study that features a docking task in VR, we found that DOF-Separation significantly improves the manipulation performance of Hand Ray, while improving that of Virtual Hand only in difficult tasks of combined translation and rotation. We suggest future XR systems to adopt DOF-Separation for input in manipulation-heavy applications, such as 3D design.

Thorbjørn Mikkelsen, Qiushi Zhou, Mathias N. Lystbæk, Yang Liu, Hans Gellersen, and Ken Pfeuffer. 2025. DOF-Separation for 3D Manipulation in XR: Understanding Finger-Wrist Separation to Simultaneously Translate and Rotate Objects. In 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Daejeon, Republic of Korea. IEEE, 1246–1256.

Overview of the SpatialPad interaction technique. Left: when inactive, the system does not respond to hand movement or finger flexion. Center: activation and calibration are performed by tapping the palm of the dominant hand with the nondominant hand, capturing the relaxed baseline pose. Right: once activated, a semi-transparent circular visual marker indicate the finger flex threshold, together with the corresponding change of color for the knuckle of the finger flexed, providing real-time feedback when flex gestures are detected.

Interacting with eyes and hands inputs is attractive for extended reality (XR). We explore how further human hand capabilities can become compatible with gaze for interaction. We introduce a gaze- and air-tap based interaction technique that enables mobile one-handed mid-air gestures for XR. By reimagining trackpad multi-touch gestures in 3D space as multi-air-taps based on a hand-attached control interface, our technique allows users to instantly access shortcuts to commands like object dragging, scrolling, and window switching. We present a prototype application that explores the capability for expressive input in compound object manipulation and UI navigation tasks. Our insights highlight our novel technique’s potential as a natural and efficient input method, expanding the interaction repertoire of extended reality users.

Juan Sánchez Esquivel, Lauren Zerbin, Qiushi Zhou, Ken Pfeuffer, and Hans Gellersen. 2025. SpatialPad – Air-taps for interaction in XR. In 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Daejeon, Republic of Korea. IEEE, 40–45.

In 3D user interfaces, reaching out to grab and manipulate something works great until it is out of reach. Indirect techniques like gaze and pinch offer an alternative for distant interaction, but do not provide the same immediacy or proprioceptive feedback as direct gestures. To support direct gestures for faraway objects, we introduce SightWarp: an interaction technique that exploits eye-hand coordination to seamlessly summon object proxies to the user’s fingertips. The idea is that after looking at a distant object, users either shift their gaze to the hand or move their hand into view—triggering the creation of a scaled near-space proxy of the object and its surrounding context. The proxy remains active until the eye–hand pattern is released. The key benefit is that users always have an option to immediately operate on the distant object through a natural, direct hand gesture. Through a user study of a 3D object docking task, we show that users can easily employ SightWarp, and that subsequent direct manipulation improves performance over gaze and pinch. Application examples illustrate its utility for 6DOF manipulation, overview-and-detail navigation, and world-in-miniature interaction. Our work contributes to expressive and flexible object interactions across near and far spaces.

Yang Liu, Thorbjørn Mikkelsen, Zehai Liu, Gengchen Tian, Diako Mardanbegi, Qiushi Zhou, Hans Gellersen, and Ken Pfeuffer. 2025. At a Glance to Your Fingertips: Enabling Direct Manipulation of Distant Objects Through SightWarp. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25), Busan, Republic of Korea. Association for Computing Machinery, New York, NY, USA, Article 26, 12 pages.

Overview of earEOG headphones showing the electrode positions and study apparatus. (a) The EOG headphones have a standard headphone form factor with 8 gold-plated electrodes (copper base) on each ear (One reference (R) and one Ground (G) electrode). The left headphone includes the Cyton Board while the right headphone includes the battery; (b) Schematic drawing showing the positioning of the electrodes around the ears; (c) A study participant wearing the EOG headphones and gold-standard Electrooculography simultaneously.

Eye tracking technology is frequently utilized to diagnose eye and neurological disorders, assess sleep and fatigue, study human visual perception, and enable novel gaze-based interaction methods. However, traditional eye tracking methodologies are constrained by bespoke hardware that is often cumbersome to wear, complex to apply, and demands substantial computational resources. To overcome these limitations, we investigated the application of Electrooculography (EOG) eye tracking using 14 electrodes positioned around the ears, integrated into a custom-built headphone form factor device. In a controlled laboratory experiment, 16 participants tracked a series of on-screen stimuli designed to induce smooth pursuits and saccades. Data analysis identified the optimal electrode pairs for tracking vertical and horizontal eye movements, benchmarked against gold-standard EOG and camera-based eye tracking. The electrode montage closest to the eyes provided the best results for horizontal eye movements. One-dimensional smooth pursuit eye movements measured via earEOG exhibited a high correlation with the gold-standard for horizontal 1D pursuits spanning 2.5° to 15° visual angle for the best performing electrode pair (r_EOG = 0.81, p=0.01; r_CAM = 0.56, p=0.02). Vertical 1D smooth pursuits were only weakly correlated for the best performing pair (r_EOG = 0.32, p=0.03; r_CAM = 0.31, p=0.05). Voltage deflections of earEOG and gold-standard EOG for saccades from 2.5° to 15° in the four cardinal directions are highly correlated for horizontal eye movement (r_left = 0.99, p<0.001; r_right = 1.0, p<0.001) but not for vertical eye movements (r_up = 0.79, p=0.06; r_down = 0.35, p=0.53). A regression model was employed to predict absolute gaze angle changes of horizontal saccades using earEOG and gold-standard EOG. In the left and right directions, the earEOG model achieved a mean absolute angular error of 3.99 ° ± 3.45 °, for saccades ranging from 2.5° to 15°. In comparison, gold-standard EOG attained mean absolute angular error of 2.98 ° ± 2.44 °. Overall, horizontal earEOG demonstrated strong performance, indicating its potential effectiveness in our setup. On the other hand, vertical earEOG showed significantly poorer results, suggesting that it may not be feasible with our current configuration.

Tobias King, Michael Knierim, Philipp Lepold, Christopher Clarke, Hans Gellersen, Michael Beigl, and Tobias Röddiger. 2025. earEOG via periauricular electrodes to facilitate eye tracking in a natural headphone form factor. Scientific Reports 15, 32437.

Feasibility evaluation of BodyPursuits as HCI method. (A) Setup: User study participant performing BodyPursuits gesture, wearing Pupil Core (mobile eye tracking hardware by Pupil Labs) [Kassner et al. 2014] and Ticwatch E2 as baseline. (B) Evaluation: Gesture classifier training and performance evaluation, as well as user experience evaluation.

Smooth pursuits are natural eye movements that occur when we track a moving target with our gaze. While they were explored as a gaze-based input method using external screens to display moving stimuli, we propose BodyPursuits, a novel HCI method that eliminates additional screens. Stimuli are generated by users tracing smooth trajectories with their hand in mid-air while fixating their gaze on their thumb. We conducted a user study to collect eye-tracking and baseline IMU data from 20 participants performing 10 BodyPursuits gestures. Based on 1800 samples and noise data, we train a TinyHAR classifier. It achieves a macro-average F1 score of 0.772. With UEQ results indicating a positive user experience and RTLX scores showcasing low subjective workload for all 10 gestures, we successfully demonstrated BodyPursuits’ potential as viable interaction method. We envision BodyPursuits could be integrated into EOG earphones to detect mid-air hand gestures without external screens or cameras.

Anja Hansen, Sarah Makarem, Kai Kunze, Yexu Zhou, Michael Thomas Knierim, Christopher Clarke, Hans Gellersen, Michael Beigl, and Tobias Röddiger. 2025. BodyPursuits: Exploring Smooth Pursuit Gaze Interaction Based on Body Motion Targets. In Proceedings of the 2025 Symposium on Eye Tracking Research and Applications (ETRA '25), Tokyo, Japan. Association for Computing Machinery, New York, NY, USA, Article 3, 8 pages.

Implicit eye tracking calibration algorithm proposed for Online-EYE using multimodal input UI selection. In this work, we contribute an evaluation of the approach with controllers in VR, but the principle remains the same and can be extended to bare hands, mouse, and other input modalities for interaction, in any XR device with known UI positions.

Unlike other inputs for extended reality (XR) that work out of the box, eye tracking typically requires custom calibration per user or session. We present a multimodal inputs approach for implicit calibration of eye tracker in VR, leveraging UI interaction for continuous, background calibration. Our method analyzes gaze data alongside controller interaction with UI elements, and employing ML techniques it continuously refines the calibration matrix without interrupting users from their current tasks. Potentially eliminating the need for explicit calibration. We demonstrate the accuracy and effectiveness of this implicit approach across various tasks and real time applications achieving comparable eye tracking accuracy to native, explicit calibration. While our evaluation focuses on VR and controller-based interactions, we anticipate the broader applicability of this approach to various XR devices and input modalities.

Baosheng James Hou, Lucy Abramyan, Prasanthi Gurumurthy, Haley Adams, Ivana Tosic Rodgers, Eric J Gonzalez, Khushman Patel, Andrea Colaço, Ken Pfeuffer, Hans Gellersen, Karan Ahuja, and Mar Gonzalez-Franco. 2025. Online-EYE: Multimodal Implicit Eye Tracking Calibration for XR. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25), Yokohama, Japan. Association for Computing Machinery, New York, NY, USA, Article 550, 16 pages.

The On-body Icons interface enables users to quickly launch apps in AR and offers onboarding by letting them experience the feature firsthand (a-c) and guiding them through the setup of on-body shortcuts (d, f). It also supports methods for setting up shortcuts beyond "grab-and-place" (e), such as "select-and-point" (g).

On-body tapping provides a quick way to launch augmented reality (AR) apps using virtual shortcuts placed on the user’s skin, clothes, and jewelry. While prior work has focused on tapping performance, social acceptance, and sensing techniques, users’ behaviour in placing shortcuts on their body has been underexplored. In this work, we propose On-body Icons — a novel interface for launching apps via touching virtual icons placed across the user’s entire body, and use it to investigate locations, reasons for chosen icon placement, and users’ attitudes towards the feature. Results of the qualitative study conducted with 24 participants demonstrated that people employ a wide variety of placement strategies that balance memorability of the locations with accuracy and comfort of reaching the icons. We discuss these findings in regard to current understanding of memorability of icon placement, placement appropriateness, and privacy, and offer design implications for similar features in spatial applications.

Uliana Tsimbalistaia, Caroline Berger, Hans Gellersen, and Pavel Manakhov. 2025. On-body Icons: Designing a 3D Interface for Launching Apps in Augmented Reality. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25), Yokohama, Japan. Association for Computing Machinery, New York, NY, USA, Article 629, 15 pages.

The experimental setup for investigating eye dominance in virtual reality (VR). A participant, wearing a VR headset, aligns a virtual cursor with a distant target using a controller. The two insets display the rendering for the left and right eye views, demonstrating target-controller alignment with the right eye only. In this trial, the participant is right-eye dominant.

Understanding eye dominance, the subconscious preference for one eye, has significant implications for 3D user interfaces in VR and AR, particularly in interface design and rendering. Although HCI recognizes eye dominance, little is known about what causes it to switch from one eye to another. To explore this, we studied eye dominance in VR, where 28 participants manually aligned a cursor with a distant target across three tasks. We manipulated the horizontal viewing angle, the hand used for alignment, and eye movement induced by target behaviour. Our results confirm the dynamic nature of eye dominance, though with fewer switches than expected and varying influences across tasks. This highlights the need for adaptive HCI techniques, which account for shifts in eye dominance in system design, such as gaze-based interaction, visual design, or rendering, and can improve accuracy, usability, and experience.

Franziska Prummer, Mohamed Shereef Abdelwahab, Florian Weidner, Yasmeen Abdrabou, and Hans Gellersen. 2025. It’s Not Always the Same Eye That Dominates: Effects of Viewing Angle, Handedness and Eye Movement in 3D. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25), Yokohama, Japan. Association for Computing Machinery, New York, NY, USA, Article 748, 10 pages.

Head pointing is widely used for hands-free input in head-mounted displays (HMDs). The primary role of head movement in an HMD is to control the viewport based on absolute mapping of head rotation to the 3D environment. Head pointing is conventionally supported by the same 1:1 mapping of input with a cursor fixed in the centre of the view, but this requires exaggerated head movement and limits input granularity. In this work, we propose to adopt dynamic gain to improve ergonomics and precision and introduce the HeadShift technique. The design of HeadShift is grounded in natural eye-head coordination to manage control of the viewport and the cursor at different speeds. We evaluated HeadShift in a Fitts’ Law experiment and on three different applications in VR, finding the technique to reduce error rate and effort. The findings are significant as they show that gain can be adopted effectively for head pointing while ensuring that the cursor is maintained within a comfortable eye-in-head viewing range.

Haopeng Wang, Ludwig Sidenmark, Florian Weidner, Joshua Newn, and Hans Gellersen. 2025. HeadShift: Head Pointing with Dynamic Control-Display Gain. ACM Transactions on Computer-Human Interaction 32, 1, Article 2 (2025), 28 pages.

H2E proposes a cascade of eye and head combined by a hand gesture vocabulary to enable fine grained interactions.

Eye-based interaction techniques for extended reality, such as gaze and pinch, are simple to use however suffer from input precision issues. We present H2E, an integrated fine and coarse-grained pointing framework that cascades Hand, Head, and Eye inputs. We further introduce the MagicPinch gesture as an example of the framework, which allows for smooth transitioning between fine and coarse-grained pointing. When combined together, after users initiate a pinch gesture, a cursor appears midway during the pinch at the position of the gaze, which can be dragged by head pointing if needed before pinch confirmation. This has the advantage that it can add a precision component without changing the semantics of the technique. In this paper, we describe the design of the H2E framework and implementation of the MagicPinch technique. Furthermore, we present an evaluation of our method in a Fitts-based user study, exploring the speed-accuracy trade-offs against a gaze and pinch interaction baseline.

Khushman Patel, Vrushank Phadnis, Eric J Gonzalez, Hans Gellersen, Ken Pfeuffer, and Mar Gonzalez-Franco. 2025. H2E: Hand, Head, Eye a Multimodal Cascade of Natural Inputs. In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France. IEEE, 84–89.

a)–c): Traditional eye dominance tests. d) Our virtual hole-in-card eye-dominance test. A card with a hole is attached to a controller and the participant has to look through the card at the target. This is only possible with one eye — the dominant eye.

Eye dominance, the subconscious preference for one eye, is an increasingly critical factor in optimizing user experiences in virtual and augmented reality (AR/VR) applications. However, traditional methods for determining eye dominance often produce inconsistent results, creating challenges for calibration and the development of reliable eye-dominance-based technologies. To address this, we investigated eye dominance by comparing three traditional tests with a novel in-HMD test, designed specifically for immersive AR/VR settings. Our findings revealed that while traditional methods often produced inconsistent results, the in-HMD test demonstrated greater alignment with established methods. Based on these results, we provide recommendations on testing eye dominance in future research to improve reliability. Furthermore, the in-HMD test demonstrates strong potential as a practical and valid alternative for AR/VR applications, offering a streamlined approach for calibration and advancing the development of eye-dominance-based technologies.

Franziska Prummer, Florian Weidner, and Hans Gellersen. 2025. Advancing Eye Dominance Testing: Comparing Traditional Methods with an In-HMD Approach for AR/VR Applications. In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France. IEEE, 112–116.

Conceptual illustration of dynamic eye dominance-guided foveated rendering. a) Depicts both eyes’ views with the left eye as dominant, rendered at a lower foveation level. b) Shows the shift in eye dominance following a rightward gaze shift, with foveated rendering adjusting to the newly dominant eye.

Foveated rendering is a promising technology in virtual and augmented reality (VR/AR), leveraging eye tracking to optimize computational load by rendering high-quality imagery only in the user’s gaze direction. Recent advancements have explored using eye dominance, a user’s subconscious preference for one eye, to enhance efficiency. So far, methods have treated eye dominance as a fixed trait. Evidence from vision science and psychology suggests it is dynamic and context-dependent. This paper reviews existing research on eye dominance behaviour and its variability, highlighting its implications for VR/AR rendering. Building on these insights, we propose refinements to foveated rendering, incorporating calibration and real-time prediction mechanisms to account for eye dominance variability. By embracing the dynamic nature of eye dominance, these advancements aim to optimize computational performance while maintaining a seamless and personalized user experience in VR/AR applications.

Franziska Prummer, Florian Weidner, and Hans Gellersen. 2025. All But Static: Exploring Dynamic Eye Dominance for Foveated Rendering. In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France. IEEE, 376–380.

Gaze and hand-based micro-gestures have been widely studied individually, each offering unique advantages as input modalities, but their combination remains under-explored. In this work, we integrate gaze with thumb-to-finger micro-gestures, discussing their complementary strengths to enable expressive and efficient interactions. We examine two strategies for combining gaze and microgestures, assess their strengths and weaknesses, and present examples such as mouse emulation and application shortcuts. Finally, we outline the potential and challenges of gaze and micro-gestures to enhance interaction across diverse contexts, and discuss future directions. This work begins the exploration of gaze and micro-gestures for novel input techniques.

Kagan Taylor, Haopeng Wang, Florian Weidner, and Hans Gellersen. 2025. Towards the Fusion of Gaze and Micro-Gestures. In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France. IEEE, 1474–1475.

Eye tracking filters have been shown to improve accuracy of gaze estimation and input for stationary settings. However, their effectiveness during physical movement remains underexplored. In this work, we compare common online filters in the context of physical locomotion in extended reality and propose alterations to improve them for on-the-go settings. We conducted a computational experiment where we simulate performance of the online filters using data on participants attending visual targets located in world-, path-, and two head-based reference frames while standing, walking, and jogging. Our results provide insights into the filters' effectiveness and factors that affect it, such as the amount of noise caused by locomotion and differences in compensatory eye movements, and demonstrate that filters with saccade detection prove most useful for on-the-go settings. We discuss the implications of our findings and conclude with guidance on gaze data filtering for interaction in extended reality.

Pavel Manakhov, Ludwig Sidenmark, Ken Pfeuffer, and Hans Gellersen. 2024. Filtering on the Go: Effect of Filters on Gaze Pointing Accuracy During Physical Locomotion in Extended Reality. IEEE Transactions on Visualization and Computer Graphics 30, 11 (2024), 7234–7244.

Schematic view of how signal processing could detect and filter motor tics. VR and AR often rely on head-, eye- and hand-input for control. Tic disorders, especially motor tics, disrupt this traditional AR/VR control cycle. Adjusting AR/VR, e.g. via signal processing — is a potential way to make immersive technologies enjoyable and usable for people with certain manifestations of tic disorders.

Tics — sudden and irresistible movements, sounds, or behaviour of people — can severely impact people’s lives. Interactive technologies such as virtual reality (VR) and augmented reality (AR) can potentially act as entertainment and productivity technologies that can handle tics. However, current AR/VR technologies are not ready for many people with tics as the tic itself — especially motor tics — prevents usage as many user interfaces rely on hand-, head- and eye-interaction. However, these body parts often affect motor tics, preventing an enjoyable, immersive experience. In this position paper, we outline the basics of tic disorders, connect tic disorders to head-mounted displays, and outline challenges and opportunities. By scoping literature and relying on anecdotal data, we speculate on whether and how we can design immersive applications and technologies for people with tic disorders. We stress that working with users and the availability of public data sets is essential for research and development. The prediction, filtering, or leveraging of tic-induced motion patterns especially seems to be a promising research avenue. In addition, we outline potential strategies such as input remapping and modifying the audiovisual output of the HMD could be improved to prevent or inhibit tics. Overall, we open the discussion about the design of immersive experiences for people with tic disorders.

Joseph McVelia, Florian Weidner, and Hans Gellersen. 2024. VR/AR for Users with Tic Disorders: Challenges and Opportunities. In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Bellevue, WA, USA. IEEE, 239–242.

Example of using Gaze&Wall, a novel interaction technique designed to enable precise object selection in dense environments. Users initially focus their gaze on the desired object. To tackle occlusion, users can indicate their desired depth by pointing with their dominant hand (DH) towards the ground. They finalize the selection with a pinch gesture with the non-dominant hand (NDH).

Raypointing, the status-quo pointing technique for virtual reality, is challenging with many occluded and overlapping objects. In this work, we investigate how eye-tracking input can assist the gestural ray pointing in the disambiguation of targets in densely populated scenes. We explore the concept of Gaze + Plane, where the intersection between the user’s gaze and a hand-controlled plane facilitates 3D position specification. In particular, two techniques are investigated: Gaze&Wall, which employs an indirect plane positioned in depth using a hand ray, and Gaze&Racket, featuring a hand-held and rotatable plane. In a first experiment, we reveal the speed-error trade-offs between Gaze + Plane techniques. In a second study, we compared the best techniques to newly designed gesture-only techniques, finding that Gaze&Wall is less error-prone and significantly faster. Our research has relevance for spatial interaction, specifically on advanced techniques for complex 3D tasks.

Uta Wagner, Matthias Albrecht, Andreas Asferg Jacobsen, Haopeng Wang, Hans Gellersen, and Ken Pfeuffer. 2024. Gaze, Wall, and Racket: Combining Gaze and Hand-Controlled Plane for 3D Selection in Virtual Reality. Proceedings of the ACM on Human-Computer Interaction 8, ISS, Article 534 (2024), 25 pages.

Extended Reality (XR) systems with hand-tracking support direct manipulation of objects with both hands. A common interaction in this context is for the non-dominant hand (NDH) to orient an object for input by the dominant hand (DH). We explore bimanual interaction with gaze through three new modes of interaction where the input of the NDH, DH, or both hands is indirect based on Gaze+Pinch. These modes enable a new dynamic interplay between our hands, allowing flexible alternation between and pairing of complementary operations. Through applications, we demonstrate several use cases in the context of 3D modelling, where users exploit occlusion-free, low-effort, and fluid two-handed manipulation. To gain a deeper understanding of each mode, we present a user study on an asymmetric rotate-translate task. Most participants preferred indirect input with both hands for lower physical effort, without a penalty on user performance. Otherwise, they preferred modes where the NDH oriented the object directly, supporting preshaping of the hand, which is more challenging with indirect gestures. The insights gained are of relevance for the design of XR interfaces that aim to leverage eye and hand input in tandem.

Mathias N. Lystbæk, Thorbjørn Mikkelsen, Roland Krisztandl, Eric J Gonzalez, Mar Gonzalez-Franco, Hans Gellersen, and Ken Pfeuffer. 2024. Hands-on, Hands-off: Gaze-Assisted Bimanual 3D Interaction. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24), Pittsburgh, PA, USA. Association for Computing Machinery, New York, NY, USA, Article 80, 12 pages.

Hand-tracking in Extended Reality (XR) enables moving objects in near space with direct hand gestures, to pick, drag and drop objects in 3D. In this work, we investigate the use of eye-tracking to reduce the effort involved in this interaction. As the eyes naturally look ahead to the target for a drag operation, the principal idea is to map the translation of the object in the image plane to gaze, such that the hand only needs to control the depth component of the operation. We have implemented four techniques that explore two factors: the use of gaze only to move objects in X-Y vs. extra refinement by hand, and the use of hand input in the Z axis to directly move objects vs. indirectly via a transfer function. We compared all four techniques in a user study (N=24) against baselines of direct and indirect hand input. We detail user performance, effort and experience trade-offs and show that all eye-hand techniques significantly reduce physical effort over direct gestures, pointing toward effortless drag-and-drop for XR environments.

Uta Wagner, Andreas Asferg Jacobsen, Tiare Feuchtner, Hans Gellersen, and Ken Pfeuffer. 2024. Eye-Hand Movement of Objects in Near Space Extended Reality. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24), Pittsburgh, PA, USA. Association for Computing Machinery, New York, NY, USA, Article 84, 13 pages.

Basic operation of Gaze + Pinch in contrast to direct manipulation, demonstrating how similar (and, thus, easy to learn) the gestures are, as well as how the eyes extend reach to any object the user sees.

For Extended Reality (XR) headsets, a key aim is the natural interaction in 3-D space beyond what traditional methods of keyboard, mouse, and touchscreen can offer. With the release of the Apple Vision Pro, a novel interaction paradigm is now widely available where users seamlessly navigate content through the combined use of their eyes and hands. However, blending these modalities poses unique design challenges due to their dynamic nature and the absence of established principles and standards. In this article, we present five design principles and issues for the Gaze + Pinch interaction technique, informed by eye-hand research in the human–computer interaction field. The design principles encompass mechanisms like division of labor and minimalistic timing, which are crucial for usability, alongside enhancements for the manipulation of objects, indirect interactions, and drag & drop. Whether in design, technology, or research domains, this exploration offers valuable perspectives for navigating the evolving landscape of 3-D interaction.

Ken Pfeuffer, Hans Gellersen, and Mar Gonzalez-Franco. 2024. Design Principles and Challenges for Gaze + Pinch Interaction in XR. IEEE Computer Graphics and Applications 44, 3 (2024), 74–81.