Hand-eye Alignment — GEMINI ERC

Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality

Mathias Lystbæk, Peter Rosenberg, Ken Pfeuffer, Jens Emil Grønbæk and Hans Gellersen

ETRA ‘22: ACM Symposium on Eye Tracking Research and Applications

Gaze and freehand gestures suit Augmented Reality as users can interact with objects at a distance without need for a separate input device. We propose Gaze-Hand Alignment as a novel multimodal selection principle, defined by concurrent use of both gaze and hand for pointing and alignment of their input on an object as selection trigger. Gaze naturally precedes manual action and is leveraged for pre-selection, and manual crossing of a pre-selected target completes the selection. We demonstrate the principle in two novel techniques, Gaze&Finger for input by direct alignment of hand and finger raised into the line of sight, and Gaze&Hand for input by indirect alignment of a cursor with relative hand movement. In a menu selection experiment, we evaluate the techniques in comparison with Gaze&Pinch and a hands-only baseline. The study showed the gaze-assisted techniques to outperform hands-only input, and gives insight into trade-offs in combining gaze with direct or indirect, and spatial or semantic freehand gestures.

Mathias N. Lystbæk, Peter Rosenberg, Ken Pfeuffer, Jens Emil Grønbæk, and Hans Gellersen. 2022. Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality. Proc. ACM Hum.-Comput. Interact. 6, ETRA, Article 145 (May 2022), 18 pages.

PDF

DOI

BibTex

@article{10.1145/3530886,
	author = {Lystbæk, Mathias N. and Rosenberg, Peter and Pfeuffer, Ken and Grønbæk, Jens Emil and Gellersen, Hans},
	title = {Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality},
	year = {2022},
	publisher = {Association for Computing Machinery},
	address = {New York, NY, USA},
	url = {https://doi.org/10.1145/3530886},
	doi = {10.1145/3530886},
	booktitle = {Proc. ACM Hum.-Comput. Interact},
	articleno = {148},
	numpages = {18},
	keywords = {eye-tracking, gaze interaction, pointing, mid-air gestures, augmented reality, menu selection},
	series = {ETRA}
}