Gaze and Intention
Joshua Newn
The University of Melbourne
I'm Joshua, a PhD Candidate at the Microsoft Research Centre for Social NUI at The University of Melbourne under the supervision of Professor Frank Vetere and Dr Eduardo Velloso. I am towards the end of my candidature and am actively looking for opportunities.
My current research aims to advance the use of eye tracking to enrich the interaction between humans and machines, particularly in collaborative contexts. Humans during face-to-face interaction naturally monitor the gaze of one another as it offers a powerful 'window into the mind'. For the interaction between humans and machines be effective, machines need first to understand human gaze behaviour so that they can discern that higher cognitive functions such as intention.
I am involved in a number of projects at the university, including Biometric Mirror and Multimodal Human-Agent Collaboration. I was formerly employed as the Usability Lab Manager for the Interaction Design Lab at the university where I support a wide range of HCI research projects and UX evaluations internally and externally.
I previously completed a Master of Information Systems (Research Specialisation) at the university which focuses on multimodal interaction, particulary the combination of eye tracking and touch(+less) gestures on large interactive surfaces under the supervision of Professor Frank Vetere and Dr Marcus Carter (now at the University of Sydney). My current research interests are in the applications of novel inputs in human-computer interaction, the design of explainable interfaces for explainable AI, and the use of gaze as an input in technology-supported interaction.
Awards
ACM Transaction on Computer-Human Interaction 2017 Best Paper Award
Awarded 2019
Google PhD Travel Scholarship
Awarded 2018 and 2019
Melbourne School of Engineering Travelling Scholarship
Awarded 2016 and 2017
University of Melbourne CIS Doctoral Colloquium Best Poster Award
Awarded 2017
Melbourne School of Engineering Exchange Scholarship
Awarded 2017
Microsoft Research Centre for Social NUI Postgraduate Top-Up Scholarship
Awarded 2016
Australian Postgraduate Award
Awarded 2016
Dean's Honours List 2015
Awarded 2016
Education
Doctor of Philosophy (Engineering)
Field of Research: Human-Computer Interaction
Microsoft Research Centre for Social Natural User Interfaces,
Computing and Information Systems,
The University of Melbourne
2016 - Present
Master of Information Systems (Research Specialisation)
Field of Research: Human-Computer Interaction
Microsoft Research Centre for Social Natural User Interfaces,
Computing and Information Systems,
The University of Melbourne
2014 - 2015
Bachelor of Information Technology and Systems
Faculty of Information Technology,
Monash University
2010 - 2012
Teaching
I have been employed as an academic tutor for a number of Human-Computer Interaction and Information Systems subjects since 2016, at undergraduate and postgraduate level, with mandatory professional development training undertaken. Subjects over the years include:
Interaction Design and Usability (ISYS90085)
Evaluating the User Experience (INFO90004)
Usability Evaluation Methods (INFO20004)
Designing Novel Interactions (INFO90003)
Usability Engineering (INFO30004)
I have co-supervised nine graduate students on their thesis projects till completion, and currently co-supervising two HCI graduate student. Further, I have mentored PhD students at the beginning of their candidature, leading to the current involvement in their PhD projects.
Academic Service
I have been of service to the HCI community over the past five years. I have mainly reviewed SIGCHI and IEEE publications, including top-tier venues such as CHI, ETRA, and DIS. I have also taken part in organising committees such as the role of Accessibility Chair for CHI Play 2018 and 2019. I have also been a student volunteer at multiple conferences that I have attended, most recently at UbiComp 2019.
As an active long standing member of the HCI group at The University of Melbourne, I am knowledgeable in a wide range of topics in HCI, and have past and ongoing internal and external research collaborations.
Journals
Abdelrahman, Y., Khan, A., Newn, J., Velloso, E., Safwat, S., Bailey, J., Bulling, A., Vetere, F. & Schmidt, A. (2019). Classifying Attention Types with Thermal Imaging and Eye Tracking. In Proceedings of ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). 3, 3, Article 69 (September 2019). [DOI]
Srivastava, N., Newn, J. & Velloso, E. (2018). Combining Low and Mid-Level Gaze Features for Desktop Activity Recognition.. In Proceedings of ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). 2, 4, Article 189 (December 2018). [DOI]
Velloso, E., Carter, M., Newn, J., Esteves, A., Clarke, C. & Gellersen, H. (2017). Motion Correlation: Selecting Objects by Matching Their Movement. In ACM Transactions on Computer-Human Interaction (TOCHI). 24, 3, Article 22 (April 2017). [DOI]
TOCHI Best Paper Award 2017
Peer-Reviewed Conference Papers
Babaei, E., Srivastava, N., Newn, J., Zhou, Q., Dingler, T., & Velloso, E. (Forthcoming). Faces of Focus: A Study on the Facial Cues of Attentional States. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, NY, USA. [DOI]
Newn J., Singh, R., Velloso, E., & Vetere, F. (2019). Combining Implicit Gaze and AI for Real-Time Intention Projection. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers (UbiComp '19 Adjunct). [DOI]
Newn, J., Singh, R., Allison, F., Madumal, P., Velloso, E., & Vetere, F. (2019). Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents. In 17th IFIP International Conference on Human-Computer Interaction — INTERACT 2019 (INTERACT '19). [DOI]
Wouters, N., Kelly, R., Velloso, E., Wolf, K., Ferdous, HS., Newn, J., Joukhadar, Z., & Vetere, F. (2019). Biometric Mirror: Exploring Ethical Opinions towards Facial Analysis and Automated Decision-Making. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS '19). ACM, New York, NY, USA, 447-461. [DOI]
Zhou, Q., Newn, J., Srivastava, N., Goncalves, J., Dingler, T., & Velloso, E. (2019). Cognitive Aid: Task Assistance Based on Mental Workload Estimation. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, New York, NY, USA, Paper LBW2315, 6 pages. [DOI]
Allison, F., Newn, J., Smith, W., Gibbs, M., & Carter, M. (2019). Frame Analysis of Voice Interaction in Gameplay. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Paper 393, 14 pages. [DOI]
Singh, R., Miller, T., Newn, J., Sonenberg, L., Velloso, E., & Vetere, F. (2018). Combining Planning with Gaze for Online Human Intention Recognition. In Proceedings of the 17th Conference on Autonomous Agents and MultiAgent Systems (AAMAS '18). Richland, SC. [URL]
Newn, J., Allison, F., Velloso, E., & Vetere, F. (2018). Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA. [DOI]
Newn, J. (2018). Enabling Intent Recognition Through Gaze Awareness. In Proceedings of the 2018 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA. [DOI]
Newn, J., Velloso, E., Allison, F., Abdelrahman, Y. & Vetere, F. (2017). Evaluating Real-Time Gaze Representations to Infer Intention in Competitive Turn-Based Strategy Games. In Proceedings of the 2017 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17). ACM, New York, NY, USA, 541–552. [DOI]
Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 251–260. [DOI] [PDF]
Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Exploring the Effects of Gaze Awareness on Multiplayer Gameplay . In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (CHI PLAY Companion '16). ACM, New York, NY, USA, 239–244. [DOI] [PDF] [Poster]
Carter, M., Newn, J., Velloso, E., & Vetere, F. (2015). Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '15). ACM, New York, NY, USA, 167–176. [DOI]
Workshop papers
Accepted Submissions
Newn J., Tag, B., Singh, R., Velloso, E., & Vetere, F. (2019). AI-Mediated Gaze-Based Intention Recognition for Smart Eyewear: Opportunities & Challenges. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers (UbiComp '19 Adjunct). London, United Kingdom. [DOI]
Zhou, Q., Newn J., Tag, B., Lee, HP., Wang, C., & Velloso, E. (2019). Ubiquitous Smart Eyewear Interactions using Implicit Sensing and Unobtrusive Information Output. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers (UbiComp '19 Adjunct). London, United Kingdom. [DOI]
Newn, J., Singh, R., Allison, F., Madumal, P., Velloso, E. & Vetere, F. (2019). Nonverbal Communication in Human-AI Interaction. In INTERACT '19 Workshop on Human(s) in the Loop – Bringing AI & HCI Together. Paphos, Cyprus.
Newn, J., Singh, R., Madumal, P., Velloso, E. & Vetere, F. (2018). Designing Explainable AI Interfaces through Interaction Design Techniques. In OzCHI '18 Workshop on Interaction Design for Explainable AI. Melbourne, Australia.
Matthews, O., Sarsenbayeva, Z., Jiang, W., Newn, J., Velloso, E., Clinch, S. & Goncalves, J. (2018). Inferring the Mood of a Community From Their Walking Speed: A Preliminary Study. In Adjunct Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’18 Adjunct). Singapore, Singapore. [DOI]
Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Dynamically Exposing Gaze to Foster Playful Experiences in Multiplayer Gameplay. In CHI PLAY 2016 Workshop on Designing for Emotional Complexity in Games: The Interplay of Positive and Negative Affect. Austin, TX, USA. [PDF]
Accepted Proposals
Madumal, P., Singh, R., Newn, J., & Vetere, F. (2018). Interaction Design for Explainable AI. In Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI '18). Melbourne, VIC, Australia.
Lankes, M., Newn, J., Maurer, B., Velloso, E., Dechant, M. & Gellersen, H. (2018). EyePlay Revisited: Past, Present and Future Challenges for Eye-Based Interaction in Games. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (CHI PLAY Extended Abstracts '18'). Melbourne, VIC, Australia.
Presentations
Paper Presentations
Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents
2019 IFIP International Conference on Human-Computer Interaction (INTERACT '19),
Coral Beach Hotel & Resort, Coral Bay, Peyia, Cyprus.
Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay
2018 CHI Conference on Human Factors in Computing Systems (CHI '18),
Palais des Congrès de Montréal, Montréal, QC, Canada.
Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games
2017 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17),
Pakhuis de Zwijger, Amsterdam, North Holland, Netherlands.
Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze
2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '16),
Crowne Plaza Niagara Falls, Niagara Falls, ON, Canada.
The Effect of Gaze on Gameplay in Co-located Multiplayer Gaming Environments
2016 Computing and Information Systems Doctoral Colloquium,
Melbourne School of Engineering, Parkville, VIC, Australia.
Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback
2015 Annual Meeting of the Australian SIG for Computer Human Interaction (OzCHI '15),
Melbourne School of Design, Parkville, VIC, Australia.
Posters
Investigating Intent Recognition Through Gaze Awareness in User Interfaces
Best Poster Award
2017 Computing and Information Systems Doctoral Colloquium,
Sidney Myer Asia Centre, The University of Melbourne, Parkville, VIC, Australia.
Seminar Presentations
Nonverbal Communication in Human-AI Interaction: Utilising Gaze for Intention Recognition to Facilitate Collaborative Decision-Making (PhD Completion)
Interaction Design Group Seminar (31 May 2019),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.
Exploring Gaze Input in Social User Interfaces
hciLab Seminar (25 October 2017),
SimTech Building, The University of Stuttgart, Stuttgart, Germany.
Disentangling Gaze Cues in Intelligent User Interfaces: Opportunities & Challenges (PhD Confirmation)
Interaction Design Group Seminar (24 February 2017),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.
Gaze Patterns in Social Play: Insights into the Mind of Players
Interaction Design Group Seminar (19 August 2016),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.
Exploring the Combination of Eye Gaze and Gesture on Large Displays
Interaction Design Group Seminar (18 September 2015),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.