Skip to main content

Advertisement

Springer Nature Link
Log in

A comparison of natural user interface and graphical user interface for narrative in HMD-based augmented reality

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Over the years, the various mediums available for storytelling have progressively expanded, from spoken to written word, then to film, and now to Virtual Reality (VR) and Augmented Reality (AR). In 2016, the cutting-edge Head-Mounted Display (HMD) AR Microsoft HoloLens was released. However, though it has been several years, the quality of the user experience with narration using HMD-based AR technology has been rarely discussed. The present study explored interactive narrative in HMD-based AR regarding different user interfaces and their influence on users’ presence, narrative engagement and reflection. Inspired by an existing exhibition at the National Holocaust Centre and Museum in the U.K., a HoloLens narrative application, entitled The AR Journey, was developed by the authors using two different interaction methods, Natural User Interface (NUI) and Graphical User Interface (GUI), which were used to perform an empirical study. As revealed from the results of the between-subject design experiment, NUI exhibited statistically significant advantages in creating presence for users without 3D Role Playing Game (RPG) experience, and GUI was superior in creating presence and increasing narrative engagement for users with 3D RPG experience. As indicated by the results of the interviews, the overall narrative experience in HMD-based AR was acceptable, and the branching narrative design was engaging. However, HoloLens hardware issues, as well as virtuality and reality mismatch, adversely affected user experience. Design guidelines were proposed according to the qualitative results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Notes

  1. HoloLens is an AR HMD featuring a 35◦ viewing angle see-through holographic lenses (waveguides), spatial understanding by real-time 3D scanning, gaze tracking, hand gestures input, and voice recognition.

  2. A series of children’s gamebooks with the reader assuming the role of the protagonist and making choices that determine the main character’s actions and the plot’s outcome.

  3. Kindertransport was the title for historical events surrounding the British government efforts to bring Jewish children out of Nazi Germany, occupied Austria, and Czechoslovakia before the outbreak of World War II.

  4. The HoloLens Clicker (clicker for short) is the peripheral device built specifically for HoloLens 1 & 2. It is a miniature controller that lets the user click on whatever he or she is looking at, and there is a motion sensor inside to register the clicker’s up, down, left, and right motions.

  5. Vuforia is an augmented reality software development kit with world-leading image recognition technology.

References

  1. Aylett RS, Louchart S, Dias J, Paiva A, Vala M (2005) FearNot!–an experiment in emergent narrative. International workshop on intelligent virtual agents. Springer, Berlin, p 305-16

  2. Azuma RT (1997) A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6(4):355–385.

  3. Barba E, Macintyre B, Rouse R, Bolter J (2010) Thinking inside the box: Making meaning in a Handheld AR experience. IEEE International Symposium on Mixed & Augmented Reality-arts

  4. Bartsch A, Kalch A, Oliver MB (2014) Moved to think: The role of emotional media experiences in stimulating reflective thoughts. J Media Psychol 26(3):125

    Article  Google Scholar 

  5. Bartsch A (2012) Emotional gratification in entertainment experience. Why viewers of movies and television series find it rewarding to experience emotions. Media Psychol 15(3):267–302

    Article  Google Scholar 

  6. Billinghurst M, Grasset R, Seichter H (2010) Tangible interfaces for ambient augmented reality applications. Human-Centric Interfaces for Ambient Intelligence. Elsevier, Amsterdam, pp 281–302

  7. Botella CM, Juan MC, Baños RM, Alcañiz M, Guillén V, Rey B (2005) Mixing realities? An application of augmented reality for the treatment of cockroach phobia. Cyberpsychology & behavior 8(2):162–171

    Article  Google Scholar 

  8. Bowman DA, Chen J, Wingrave CA, Lucas J, Ray A, Polys NF et al (2006) New directions in 3d user interfaces. Int J Virtual Reality 5(2):3–14

    Article  Google Scholar 

  9. Brondi R, Alem L, Avveduto G, Faita C, Carrozzino M, Tecchia F et al (2015) Evaluating the impact of highly immersive technologies and natural interaction on player engagement and flow experience in games. International Conference on Entertainment Computing: Springer, Berlin, p 169-81

  10. Busselle R, Bilandzic H (2009) Measuring narrative engagement. Media Psychol 12(4):321–347. https://doi.org/10.1080/15213260903287259

    Article  Google Scholar 

  11. Busselle R, Bilandzic H Measuring Narrative Engagement. Media Psychol 12(4):321–47

  12. Carmigniani J, Furht B (2011) Augmented reality: an overview. Handbook of augmented reality. Springer, Berlin, pp 3–46

  13. Cavazza M, Martin O, Charles F, Mead SJ, Marichal X (2003) Interacting with virtual agents in mixed reality interactive storytelling. International Workshop on Intelligent Virtual Agents. Springer, Berlin, p 231-5

  14. Chinara C, Feingold G, Shanbhag A, Weiniger K (2018) ARnold: A mixed reality short film using microsoft hololens. SMPTE 2017 Annual Technical Conference and Exhibition, SMPTE 2017, p 1-12

  15. Christopoulos D, Mavridis P, Andreadis A, Karigiannis JN (2011) Using Virtual Environments to Tell the Story:“ The Battle of Thermopylae. 2011 Third International Conference on Games and Virtual Worlds for Serious Applications. IEEE, p 84-91

  16. Danilicheva P, Klimenko S, Baturin Y, Serebrov A (2009) Education in virtual worlds: Virtual storytelling. International Conference on CyberWorlds. IEEE, p 333-8

  17. Dow S, Mehta M, Lausier A, MacIntyre B, Mateas M (2006) Initial lessons from AR Façade, an interactive augmented reality drama. Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, p 28-es

  18. Evans G, Miller J, Pena MI, MacAllister A, Winer E (2017) Evaluating the Microsoft HoloLens through an augmented reality assembly application. Degraded Environments: Sensing, Processing, and Display 2017. International Society for Optics and Photonics, p 101970V

  19. Falcao C, Lemos AC, Soares M (2015) Evaluation of natural user interface: a usability study based on the leap motion device. Procedia Manuf 3:5490–5

    Article  Google Scholar 

  20. Funk M, Kritzler M, Michahelles F (2017) HoloLens is more than air Tap: natural and intuitive interaction with holograms. Proceedings of the seventh international conference on the internet of things, p 1-2

  21. Geslin E, Bouchard S, Richir S (2011) Gamers’ versus non-gamers’ emotional response in virtual reality. J Cyber Ther Rehabil 4(4):489–493

    Google Scholar 

  22. Gorisse G, Christmann O, Amato EA, Richir S (2017) First- and third-person perspectives in immersive virtual environments: presence and performance analysis of embodied users. Front Robot AI 4(33). https://doi.org/10.3389/frobt.2017.00033

  23. Gupta S, Tanenbaum JG, Tanenbaum K (2019) Shiva’s Rangoli: Tangible storytelling through diegetic interfaces in ambient environments. TEI 2019 - Proceedings of the 13th International Conference on Tangible, Embedded, and Embodied Interaction, p 65-75

  24. Hammady R, Ma M, Strathern C, Mohamad M (2019) Design and development of a spatial mixed reality touring guide to the Egyptian Museum. Multimed Tools Appl 79: 3465–3494.

  25. Hand S, Varan D (2009) Interactive stories and the audience: Why empathy is important. Comput Entertain 7(3):Article 39. https://doi.org/10.1145/1594943.1594951

  26. Hanna MG, Ahmed I, Nine J, Prajapati S, Pantanowitz L Augmented reality technology using microsoft HoloLens in anatomic pathology. Arch Pathol Lab Med.arpa. 2017-0189-OA

  27. Hololens 1 Hardware Specification. https://docs.microsoft.com/en-us/hololens/hololens1-hardware. Accessed 20120/1/31

  28. Ishii H (2008) The tangible user interface and its evolution. Commun ACM 51(6):32–36

    Article  Google Scholar 

  29. Jenkins H (2004) Game design as narrative. Computer 44:53

    Google Scholar 

  30. Jin Y, Ma M, Liu Y (2020) Interactive Narrative in Augmented Reality: An Extended Reality of the Holocaust. In: Chen JYC, Fragomeni G (eds) Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications. HCII 2020. Lecture Notes in Computer Science, vol 12191. Springer, Cham. https://doi.org/10.1007/978-3-030-49698-2_17

  31. Juan MC, Alcaniz M, Monserrat C, Botella C, Baños RM, Guerrero B (2005) Using augmented reality to treat phobias. IEEE Comput Graph Appl 25(6):31–37

    Article  Google Scholar 

  32. Kearsley G, Shneiderman B (1998) Engagement theory: A framework for technology-based teaching and learning. Educ Technol 38(5):20–23

    Google Scholar 

  33. Koenitz H, Chen KJ (2012) Genres, structures and strategies in interactive digital narratives – analyzing a body of works created in ASAPS. International Conference on Interactive Storytelling

  34. Larsen M (2018) Virtual sidekick: Secondperson POV in narrative VR. J Screenwriting 9(1):73–84. https://doi.org/10.1386/josc.9.1.73_1

    Article  Google Scholar 

  35. MacIntyre B, Bolter JD, Vaughn J, Hannigan B, Gandy M, Moreno E et al (2003) Three angry men: An augmented-reality experiment in point-of-view drama. Proc International Conference on Technologies for Interactive Digital Storytelling and Entertainment (TIDSEʼ03). Citeseer, p 230-6

  36. Marschner L, Pannasch S, Schulz J, Graupner ST (2015) Social communication with virtual agents: The effects of body and gaze direction on attention and emotional responding in human observers. Int J Psychophysiol 97(2):85–92. https://doi.org/10.1016/j.ijpsycho.2015.05.007

    Article  Google Scholar 

  37. Mayer RE (2014) Cognitive theory of multimedia learning. Cambridge University Press, Cambridge

  38. Moreno E, MacIntyre B, Bolter JD (2001) Alice’s adventures in new media: An exploration of interactive narratives in augmented reality. Conference on Communication of Art, Science and Technology, p 21-2

  39. Moser C, Fang X (2014) Narrative control and player experience in role playing games: decision points and branching narrative feedback. Springer International Publishing, Cham, pp 622–633

    Google Scholar 

  40. Moser C, Fang X (2015) Narrative structure and player experience in role-playing games. Int J Hum Comput Interact 31(2):146–156. https://doi.org/10.1080/10447318.2014.986639

    Article  Google Scholar 

  41. Murray JT (2018) Telltale hearts: encoding cinematic choice-based adventure games. University of California, Santa Cruz

  42. Nisi V, Wood A, Davenport G, Oakley I, Hopstory (2004) : An interactive, location-based narrative distributed in space and time. International Conference on Technologies for Interactive Digital Storytelling and Entertainment. Springer, Berlin, p 132-41

  43. O’Brien HL, Toms EG (2008) What is user engagement? A conceptual framework for defining user engagement with technology. J Am Soc Inform Sci Technol 59(6):938–955

    Article  Google Scholar 

  44. Pausch R, Proffitt D, Williams G (1997) Quantifying immersion in virtual reality. Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p 13-8

  45. Rafii A, Zuccarino T (2014) Method and system enabling natural user interface gestures with user wearable glasses. Google Patents

  46. Rembowska-Płuciennik M (2018) Second-person narration as a joint action. Lang Lit 27(3):159–175

    Article  Google Scholar 

  47. Riedl M, Saretto CJ, Young RM (2003) Managing interaction between users and agents in a multi-agent storytelling environment. Proceedings of the second international joint conference on Autonomous agents and multiagent systems, p 741-8

  48. Riedl MO, Stern A, Dini D, Alderman J (2008) Dynamic experience management in virtual worlds for entertainment, education, and training. International Transactions on Systems Science and Applications, Special Issue on Agent Based Systems for Human Learning 4(2):23–42

  49. Riedlinger U, Oppermann L, Prinz W (2019) Tango vs. HoloLens: A comparison of collaborative indoor AR visualisations using hand-held and hands-free devices. Multimodal Technol Interact 3(2):23

    Article  Google Scholar 

  50. Roth D, Bloch C, Schmitt J, Frischlich L, Latoschik ME, Bente G (2019) Perceived Authenticity, Empathy, and Pro-social Intentions evoked through Avatar-mediated Self-disclosures. ACM International Conference Proceeding Series, p 21-30

  51. Saldaña J (2015) The coding manual for qualitative researchers. Sage, Thousand Oaks

  52. Salter A (2016) Playing at empathy: Representing and experiencing emotional growth through Twine games. 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH). IEEE, p 1-8

  53. Sargunam SP, Moghadam KR, Suhail M, Ragan ED (2017) Guided head rotation and amplified head rotation: Evaluating semi-natural travel and viewing techniques in virtual reality. 2017 IEEE Virtual Reality (VR). IEEE, pp 19–28

  54. Shafer DM, Carbonara CP, Popova L (2011) Spatial presence and perceived reality as predictors of motion-based video game enjoyment. Presence 20(6):591–619

    Article  Google Scholar 

  55. Shilkrot R, Montfort N, Maes P (2014) nARratives of augmented worlds. IEEE International Symposium on Mixed and Augmented Reality - Media, Art, Social Science, Humanities and Design (ISMAR-MASH’D)

  56. Smith SP, Du’Mont S (2009) Measuring the effect of gaming experience on virtual environment navigation tasks. 2009 IEEE symposium on 3D user interfaces. IEEE, p 3-10

  57. Song Q, He L, Hu X (2012) To improve the interactivity of the history educational games with digital interactive storytelling. Physics Procedia 33:1798–1802

    Article  Google Scholar 

  58. Steuer J (1992) Defining virtual reality: Dimensions determining telepresence. J Commun 42:73-93.

  59. Studio A (2017) Introduction of fragments. https://www.asobostudio.com/games/fragments. Accessed 23 Jan 2021

  60. Taylor N, Kampe C, Bell K (2015). Me and Lee: Identification and the play of attraction in The Walking Dead. Game Studies. http://gamestudies.org/1501/articles/taylor.

  61. Vosmeer M, Roth C, Koenitz H (2017) Who are you? Voice-over perspective in surround video. International Conference on Interactive Digital Storytelling. Springer, Berlin, p 221-32

  62. Vuforia (2021) Area Targets. https://library.vuforia.com/features/environments/area-targets.html. Accessed 23 Jan 2021

  63. Weng J-F, Kuo H-l, Tseng S-S (2011) Interactive storytelling for elementary school nature science education. IEEE 11th International Conference on Advanced Learning Technologies. IEEE, p 336-8

  64. Wither J, Allen R, Samanta V, Hemanus J, Tsai Y-T, Azuma R et al (2010) The westwood experience: connecting story to locations via mixed reality. 2010 IEEE International Symposium on Mixed and Augmented Reality-Arts, Media, and Humanities. IEEE, p 39-46

  65. Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: A presence questionnaire. Presence 7(3):225–240

    Article  Google Scholar 

  66. Zhang L, Bowman DA, Jones CN (2019) Exploring effects of interactivity on learning with interactive storytelling in immersive virtual reality. 2019 11th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games). IEEE, p 1-8

  67. Zielke MA, Zakhidov D, Hardee G, Evans L, Lenox S, Orr N et al (2017) Developing Virtual Patients with VR/AR for a natural user interface in medical teaching. 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH). IEEE, p 1-8

  68. Zook A, Lee-Urban S, Riedl MO, Holden HK, Sottilare RA, Brawner KW (2012) Automated scenario generation: toward tailored and optimized military training in virtual environments. Proceedings of the international conference on the foundations of digital games, p 164-71

Download references

Acknowledgements

This work was funded partially by the National Key Research and Development Program of China (2018YFB1004903). We would like to thank Louise Stafford and Claudia Reese at the National Holocaust Centre and Museum for their support during this study.

Author information

Authors and Affiliations

  1. Tongji University, Shanghai, People’s Republic of China

    Yunshui Jin & Yongning Zhu

  2. Falmouth University, Cornwall, UK

    Minhua Ma

Authors
  1. Yunshui Jin
    View author publications

    Search author on:PubMed Google Scholar

  2. Minhua Ma
    View author publications

    Search author on:PubMed Google Scholar

  3. Yongning Zhu
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Yongning Zhu.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

(MP4 258 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, Y., Ma, M. & Zhu, Y. A comparison of natural user interface and graphical user interface for narrative in HMD-based augmented reality. Multimed Tools Appl 81, 5795–5826 (2022). https://doi.org/10.1007/s11042-021-11723-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11723-0

Keywords