Hand gestures have been shown to be beneficial for remote collaboration on physical tasks in several contexts, including mobile devices. However, to capture hand gestures in such collaborative tasks, existing approaches require to have stationary hardware systems or heavily instrumented mobile devices, making them unfeasible to be used in mobile contexts. In this paper, we present our exploration on MirrorTablet, a low-cost system that takes advantage of the built-in front-facing camera of a tablet to capture user hand gestures interacting on and above the tablet’s screen. The system requires little instrumentation on the tablet and can be easily attached/removed from the device, making it suitable for mobile scenarios. A user study with twenty participants on a novice-helper setup working on construction tasks showed that when using MirrorTablet, users completed unfamiliar tasks faster than using a common sketch-only interface. In addition, qualitative feedback from the participants also suggests noticeable design insights on hand visualization on mobile devices in remote collaboration on physical tasks.

Authors: Khanh-Duy Le, Kening Zhu, and Morten Fjeld