Abstract
| Original language | English |
|---|---|
| Title of host publication | UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology |
| Publisher | Association for Computing Machinery |
| Pages | 1010-1022 |
| Number of pages | 13 |
| ISBN (Print) | 978-1-4503-7514-6 |
| DOIs | |
| Publication status | Published - Oct 2020 |
| Event | 33rd Annual ACM Symposium on User Interface Software and Technology, UIST 2020 - Virtual Duration: 20 Oct 2020 → 23 Oct 2020 https://uist.acm.org/uist2020/ |
Conference
| Conference | 33rd Annual ACM Symposium on User Interface Software and Technology, UIST 2020 |
|---|---|
| Abbreviated title | UIST 2020 |
| Period | 20/10/20 → 23/10/20 |
| Internet address |
Keywords
- Assembly tutorial.
- Augmented reality
- Retargeting
- Video label
- 3D modeling
- Animation
- Computer aided design
- Object detection
- Object tracking
- Textures
- Assembly planning
- Information loss
- Object detection and tracking
- Presentation system
- Real-world objects
- Texture information
- User-orientation
- Viewpoint control
- User interfaces
Fingerprint
Dive into the research topics of 'Video-Annotated Augmented Reality Assembly Tutorials'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver
}
UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, 2020. p. 1010-1022.
Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review
TY - GEN
T1 - Video-Annotated Augmented Reality Assembly Tutorials
AU - Yamaguchi, M.
AU - Mori, S.
AU - Mohr, P.
AU - Tatzgern, M.
AU - Stanescu, A.
AU - Saito, H.
AU - Kalkofen, D.
N1 - Conference code: 164266 Cited By :13 Export Date: 14 December 2023 Funding details: Bundesministerium für Verkehr, Innovation und Technologie, BMVIT Funding details: Bundesministerium für Wissenschaft, Forschung und Wirtschaft, BMWFW Funding details: Österreichische Forschungsförderungsgesellschaft, FFG, 859208 Funding details: Austrian Science Fund, FWF, P30694 Funding details: Steirische Wirtschaftsförderungsgesellschaft, SFG Funding details: 854174 Funding details: Core Research for Evolutional Science and Technology, CREST, JPMJCR1683 Funding text 1: This work was enabled by the Competence Center VRVis, the FFG (grant 859208 - Matahari) and the Austrian Science Fund grant P30694, and partly by JST CREST under Grant JPMJCR1683, Japan. VRVis is funded by BMVIT, BMWFW, Styria, SFG and Vienna Business Agency in the scope of COMET, Competence Centers for Excellent Technologies (854174), which is managed by FFG. References: Agrawala, M., Phan, D., Heiser, J., Haymaker, J., Klingner, J., Hanrahan, P., Tversky, B., Designing effective step-by-step assembly instructions (2003) ACM Transactions on Graphics, 22 (3), pp. 828-837. , July 2003; Ayres, P., Marcus, N., Chan, C., Qian, N., Learning hand manipulative tasks: When instructional animations are superior to equivalent static representations (2009) Computers in Human Behavior, 25 (2), pp. 348-353. , March 2009; Bangor, A., Kortum, P., Miller, J., Determining what individual sus scores mean: Adding an adjective rating scale (2009) J. Usability Studies, 4 (3), pp. 114-123. , May 2009; Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B., Patchmatch: A randomized correspondence algorithm for structural image editing (2009) ACM Transactions on Graphics, 28 (3). , 2009; Biard, N., Cojean, S., Jamet, E., Effects of segmentation and pacing on procedural learning by video (2018) Computers in Human Behavior, 89, pp. 411-417. , December 2018; Breedveld, P., Observation, manipulation, and eye-hand coordination problems in minimally invasive surgery (1997) Proc. XVI European Annual Conference on Human Decision Making and Manual Control, pp. 9-11; Brooke, J., Sus: A quick and dirty usability scale (1996) Usability Evaluation in Industry, 189 (194), pp. 4-7. , 1996; Butz, A., Betty: Planning and generating animations for the visualization of movements and spatial relations (1994) Proc. Of Advanced Visual Interfaces, pp. 53-58; Caudell, T.P., Mizell, D.W., Augmented reality: An application of heads-up display technology to manual manufacturing processes (1992) Proc. Of Hawaii Int. Conf. Of Syst. Sc., 2, pp. 659-669; Damen, D., Leelasawassuk, T., Haines, O., Calway, A., Mayol-Cuevas, W., You-do, i-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video (2014) BMVC, p. 3; Mello De Homem, L.S., Sanderson, A.C., A correct and complete algorithm for the generation of mechanical assembly sequences (1991) IEEE Trans Robotics and Automation, 7 (2), pp. 228-240. , April 1991; Eckhoff, D., Sandor, C., Lins, C., Eck, U., Kalkofen, D., Hein, A., Tutar: Augmented reality tutorials for hands-only procedures (2018) ACM SIGGRAPH Int. Conf. On VR Continuum and Its Applications in Industry, pp. 1-3; Feiner, S., Parafrustum: Visualization techniques for guiding a user to a constrained set of viewing positions and orientations (2014) UIST, pp. 331-340. , 2014; Feiner, S., Macintyre, B., Seligmann, D., Knowledge-based augmented reality (1993) Commun. ACM, 36 (7), pp. 53-62. , July 1993; Fiala, M., Magic mirror system with hand-held and wearable augmentations (2007) 2007 IEEE Virtual Reality Conference, pp. 251-254. , 2007; Fritz, C.O., Morris, P.E., Richler, J.J., Effect size estimates: Current use, calculations, and interpretation (2012) Journal of Experimental Psychology: General, 141 (1), pp. 2-18. , 2012; Garland, T.B., Sanchez, C.A., Rotational perspective and learning procedural tasks from dynamic media (2013) Computers and Education, 69, pp. 31-37. , 2013; Goto, M., Yuko, U., Saito, H., Senda, S., Iketani, A., Task support system by displaying instructional video onto ar workspace (2010) ISMAR, pp. 83-90; Gupta, A., Fox, D., Curless, B., Cohen, M., Duplotrack: A real-time system for authoring and guiding duplo block assembly (2012) UIST, pp. 389-402; Hart, S.G., Staveland, L.E., Development of nasa-tlx (task load index): Results of empirical and theoretical research (1988) Advances in Psychology, 52, pp. 139-183. , 1988; Heiser, J., Phan, D., Agrawala, M., Tversky, B., Hanrahan, P., Identifcation and validation of cognitive design principles for automated generation of assembly instructions (2004) Proc. Of the Working Conference on Advanced Visual Interfaces, pp. 311-319. , ACM Press, New York, NY, USA; Henderson, S., Feiner, S., Exploring the benefts of augmented reality documentation for maintenance and repair (2011) TVCG, 17 (10), pp. 1355-1368. , October 2011; Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., Navab, N., Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes (2012) Asian Conference on Computer Vision, pp. 548-562. , Springer; Höffer, T.N., Leutner, D., Instructional animation versus static pictures: A meta-analysis (2007) Learning and Instruction, 17 (6), pp. 722-738. , December 2007; Kalkofen, D., Mendez, E., Schmalstieg, D., Comprehensible visualization for augmented reality (2009) IEEE TVCG, 15 (2), pp. 193-204. , 2009; Kalkofen, D., Tatzgern, M., Schmalstieg, D., Explosion diagrams in augmented reality (2009) VR'09, pp. 71-78; Kerbl, B., Kalkofen, D., Steinberger, M., Schmalstieg, D., Interactive disassembly planning for complex objects (2015) Computer Graphics Forum, 34, pp. 287-297. , Wiley Online Library; Langlotz, T., Zingerle, M., Grasset, R., Kaufmann, H., Reitmayr, G., Ar record replay: Situated compositing of video content in mobile augmented reality (2012) OzCHI, pp. 318-326; Ledermann, F., Schmalstieg, D., April a high-level framework for creating augmented reality presentations (2005) Proc. Of IEEE Virtual Reality, pp. 187-194; Li, C., Kitani, K.M., Pixel-level hand detection in ego-centric videos (2013) IEEE CVPR, pp. 3570-3577; Li, W., Agrawala, M., Curless, B., Salesin, D., Automated generation of interactive 3D exploded view diagrams (2008) ACM Transactions on Graphics, 27 (3), pp. 1011-1017. , 2008; Boesen Madsen, J., Tatzgern, M., Madsen, C.B., Schmalstieg, D., Kalkofen, D., Temporal coherence strategies for augmented reality labeling (2016) IEEE TVCG, 22 (4), pp. 1415-1423. , 2016; Makris, S., Pintzos, G., Rentzos, L., Chryssolouris, G., Assembly support using ar technology based on automatic sequence generation (2013) CIRP Annals, 62 (1), pp. 9-12. , 2013; Mijksenaar, P., Westendorp, P., Open here (1999) The Art of Instructional Design, , Thames & Hudson; Miller, G.A., The magical number seven, plus or minus two: Some limits on our capacity for processing information (1956) Psychological Review, 63 (2), pp. 81-97. , March 1956; Mitra, N.J., Yang, Y., Yan, D., Li, W., Agrawala, M., Illustrating how mechanical assemblies work (2010) ACM Transactions on Graphics, 29 (4), pp. 1-58. , July 2010; Mohr, P., Kerbl, B., Donoser, M., Schmalstieg, D., Kalkofen, D., Retargeting technical documentation to augmented reality (2015) ACM CHI. ACM, pp. 3337-3346. , New York, NY, USA; Mohr, P., Mandl, D., Tatzgern, M., Veas, E.E., Schmalstieg, D., Kalkofen, D., Retargeting video tutorials showing tools with surface contact to augmented reality (2017) ACM CHI. ACM, pp. 6547-6558; Mohr, P., Tatzgern, M., Langlotz, T., Lang, A., Schmalstieg, D., Kalkofen, D., Trackcap: Enabling smartphones for 3D interaction on mobile head-mounted displays (2019) ACM CHI, pp. 1-11; Nienhaus, M., Döllner, J., Dynamic glyphs-depicting dynamics in images of 3D scenes (2003) Int. Conf. On Smart Graphics, pp. 102-111. , Springer; Paas, F.G., Training strategies for attaining transfer of problem solving skills in statistics: A cognitive-load approach (1992) Journal of Educational Psychology, 84 (4), pp. 429-434. , 1992; Petersen, N., Stricker, D., Learning task structure from video examples for workfow tracking and authoring (2012) 2012 IEEE ISMAR, pp. 237-246; Pongnumkul, S., Dontcheva, M., Li, W., Wang, J., Bourdev, L., Avidan, S., Cohen, M.F., Pause-and-play: Automatically linking screencast video tutorials with applications (2011) ACM UIST, pp. 135-144; Prisacariu, V.A., Reid, I.D., Pwp3d: Real-time segmentation and tracking of 3D objects (2012) Int. J. Of Computer Vision, 98 (3), pp. 335-354. , 2012; Reiners, D., Stricker, D., Klinker, G., Müller, S., Augmented reality for construction tasks: Doorlock assembly (1999) Proc. Of the International Workshop on AR: Placing Artifcial Objects in Real Scenes: Placing Artifcial Objects in Real Scenes, pp. 31-46. , AK Peters, Ltd; Rother, C., Kolmogorov, V., Blake, A., Grabcut: Interactive foreground extraction using iterated graph cuts (2004) ACM Transactions on Graphics, 23, pp. 309-314; Shao, T., Li, D., Rong, Y., Zheng, C., Zhou, K., Dynamic furniture modeling through assembly instructions (2016) ACM Transactions on Graphics, 35 (6). , 2016; Shao, T., Li, W., Zhou, K., Xu, W., Guo, B., Mitra, N.J., Interpreting concept sketches (2013) Transactions on Graphics, 32 (4). , 2013; Spanjers, E.I.A., Van Gog, T., Wouters, P., Merriënboer Van, G.J.J., Explaining the segmentation effect in learning from animations: The role of pausing and temporal cueing (2012) Computers and Education, 59 (2), pp. 274-280. , 2012; Suzuki, S., Abe, K., Topological structural analysis of digitized binary images by border following (1985) Computer Vision, Graphics, and Image Processing, 30 (1), pp. 32-46. , 1985; Sweller, J., Merrienboer Van, G.J.J., Paas, C.W.F.G., Cognitive architecture and instructional design (1998) Educational Psychology Review, 10 (3), pp. 251-296. , 1998; Tang, A., Owen, C., Biocca, F., Mou, W., Comparative effectiveness of augmented reality in object assembly (2003) ACM CHI. ACM, pp. 73-80; Tatzgern, M., Grasset, R., Veas, E., Kalkofen, D., Seichter, H., Schmalstieg, D., Exploring real world points of interest: Design and evaluation of object-centric exploration techniques for augmented reality (2015) Pervasive and Mobile Computing, 18, pp. 55-70. , 2015; Tatzgern, M., Kalkofen, D., Grasset, R., Schmalstieg, D., Hedgehog labeling: View management techniques for external labels in 3D space (2014) Proc. Of IEEE Virtual Reality (VR), pp. 1-6; Wang, B., Wang, G., Sharf, A., Li, Y., Zhong, F., Qin, X., CohenOr, D., Chen, B., Active assembly guidance with online video parsing (2018) IEEE VR. IEEE, pp. 459-466; Wolfe, J.M., Guided search 2. 0 a revised model of visual search (1994) Psychonomic Bulletin & Review, 1, pp. 202-238. , 1994; Wong, A., Marcus, N., Ayres, P., Smith, L., Cooper, G.A., Paas, F., Sweller, J., Instructional animations can be superior to statics when learning human motor skills (2009) Computers in Human Behavior, 25 (2), pp. 339-347. , March 2009; Wu, J., Zhou, B., Russell, R., Kee, V., Wagner, S., Hebert, M., Torralba, A., Johnson, D.M.S., Real-time object pose estimation with pose interpreter networks (2018) IEEE IROS. IEEE, pp. 6798-6805; Wu, L., Lin, I., Tsai, M., Augmented reality instruction for object assembly based on markerless tracking (2016) ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, pp. 95-102; Zauner, J., Haller, M., Brandl, A., Hartmann, W., Authoring of a mixed reality assembly instructor for hierarchical structures (2003) IEEE/ACM ISMAR, pp. 237-246
PY - 2020/10
Y1 - 2020/10
N2 - We present a system for generating and visualizing interactive 3D Augmented Reality tutorials based on 2D video input, which allows viewpoint control at runtime. Inspired by assembly planning, we analyze the input video using a 3D CAD model of the object to determine an assembly graph that encodes blocking relationships between parts. Using an assembly graph enables us to detect assembly steps that are otherwise difficult to extract from the video, and generally improves object detection and tracking by providing prior knowledge about movable parts. To avoid information loss, we combine the 3D animation with relevant parts of the 2D video so that we can show detailed manipulations and tool usage that cannot be easily extracted from the video. To further support user orientation, we visually align the 3D animation with the real-world object by using texture information from the input video. We developed a presentation system that uses commonly available hardware to make our results accessible for home use and demonstrate the effectiveness of our approach by comparing it to traditional video tutorials. © 2020 ACM.
AB - We present a system for generating and visualizing interactive 3D Augmented Reality tutorials based on 2D video input, which allows viewpoint control at runtime. Inspired by assembly planning, we analyze the input video using a 3D CAD model of the object to determine an assembly graph that encodes blocking relationships between parts. Using an assembly graph enables us to detect assembly steps that are otherwise difficult to extract from the video, and generally improves object detection and tracking by providing prior knowledge about movable parts. To avoid information loss, we combine the 3D animation with relevant parts of the 2D video so that we can show detailed manipulations and tool usage that cannot be easily extracted from the video. To further support user orientation, we visually align the 3D animation with the real-world object by using texture information from the input video. We developed a presentation system that uses commonly available hardware to make our results accessible for home use and demonstrate the effectiveness of our approach by comparing it to traditional video tutorials. © 2020 ACM.
KW - Assembly tutorial.
KW - Augmented reality
KW - Retargeting
KW - Video label
KW - 3D modeling
KW - Animation
KW - Computer aided design
KW - Object detection
KW - Object tracking
KW - Textures
KW - Assembly planning
KW - Information loss
KW - Object detection and tracking
KW - Presentation system
KW - Real-world objects
KW - Texture information
KW - User-orientation
KW - Viewpoint control
KW - User interfaces
U2 - 10.1145/3379337.3415819
DO - 10.1145/3379337.3415819
M3 - Conference contribution
SN - 978-1-4503-7514-6
SP - 1010
EP - 1022
BT - UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology
PB - Association for Computing Machinery
T2 - 33rd Annual ACM Symposium on User Interface Software and Technology, UIST 2020
Y2 - 20 October 2020 through 23 October 2020
ER -