I am a Postdoctoral Researcher in Human-AI Interaction in the School of Computing and Information Systems at The University of Melbourne, where I am affiliated with the well-established Human-Computer Interaction Group and the AI and Autonomy Lab. I am seeking new employment opportunities where I can make impactful contributions.
I completed my PhD at the Microsoft Research Centre for Social Natural User Interfaces located within The University of Melbourne, under the supervision of Professor Frank Vetere and Dr Eduardo Velloso. My PhD thesis, titled Gaze-Based Intention Recognition for Human-Agent Collaboration: Towards Nonverbal Communication in Human-AI Interaction investigated how intelligent computer systems can become better collaborative partners by giving them the ability to infer the intentions of human partners through their eye gaze. The thesis presents guidelines for building intention-aware human-centred AI systems and reinforces the need to incorporate nonverbal communication in human-AI interaction.
My primary research area lies at the intersection of HCI and AI, particularly in the design of effective communication inputs and techniques to increase the collaborative capacity of AI-enabled systems. My broader research interests are in the applications of novel inputs in human- and player-computer interaction, multimodal sensing for cognition- and context-aware systems (particularly in learning and health) and the design of AI interfaces.
I am currently involved in a wide range of funded projects, including Biometric Mirror, Multimodal Human-Agent Collaboration and several projects in collaboration with the Defence Science and Technology Group (Department of Defence, Australia) mentored by Professor Tim Miller. I have held a number of teaching and professional roles in the past years, including the role of managing the state-of-the-art User Experience Lab, where I supported a wide range of HCI research projects and UX evaluations internally and externally.
Feel to explore the rest of the site, or to contact me through one of the links below. Cheers!
Stay tuned for updates! I am currently updating this page to show projects instead of research areas.
This body of research aims to advance the use of eye tracking to enrich the interaction between humans and machines, particularly in collaborative contexts. Humans during face-to-face interaction naturally monitor the gaze of one another as it offers a powerful 'window into the mind'. For the interaction between humans and machines be effective, machines need first to understand human gaze behaviour so that they can discern that higher cognitive functions such as intention.
My contributions to the field include the development of gaze-supported interaction techniques, in the design of visualisations for gaze data, in the analysis of gaze data for artificial agents, and in understanding the social dynamics that arise when multiple users can see each other's gaze.
ACM Transaction on Computer-Human Interaction 2017 Best Paper Award
Awarded 2019
Google PhD Travel Scholarship
Awarded 2018 and 2019
Melbourne School of Engineering Travelling Scholarship
Awarded 2016, 2017 and 2019
University of Melbourne CIS Doctoral Colloquium Best Poster Award
Awarded 2017
Melbourne School of Engineering Exchange Scholarship
Awarded 2017
Microsoft Research Centre for Social NUI Postgraduate Top-Up Scholarship
Awarded 2016
Australian Postgraduate Award
Awarded 2016
Dean's Honours List 2015
Awarded 2016
Doctor of Philosophy (Engineering)
Microsoft Research Centre for Social Natural User Interfaces,
School of Computing and Information Systems,
The University of Melbourne, Australia.
2016 - 2020
Master of Information Systems (Research Specialisation)
Microsoft Research Centre for Social Natural User Interfaces,
School of Computing and Information Systems,
The University of Melbourne, Australia.
2014 - 2015
Bachelor of Information Technology and Systems
Faculty of Information Technology,
Monash University, Australia.
2010 - 2012
I have been employed as an academic tutor for a number of Human-Computer Interaction and Information Systems subjects since 2016, teaching at both undergraduate and postgraduate levels, with mandatory professional development training undertaken. Subjects over the years include:
Interaction Design and Usability (ISYS90085)
Evaluating the User Experience (INFO90004)
Usability Evaluation Methods (INFO20004)
Designing Novel Interactions (INFO90003)
Usability Engineering (INFO30004)
Fundamentals of Interaction Design (INFO10003)
I have co-supervised thirteen postgraduate students on their thesis projects till completion. Further, I have mentored nine research higher degree students (7 PhD, 1 MPhil, 1 visiting student from NTCU Taiwan) as part of my academic training, leading to the current involvement in their PhD project outputs.
I have been of service to the HCI community over the past five years. I have mainly reviewed ACM SIGCHI publications, including top-tier venues such as CHI, CSCW, UIST, ETRA, CHI Play and DIS. I have received 3 Special Recognitions for Outstanding Reviews (CHI2019, CHI2021, CSCW2021) for my service. I have also taken part in organising committees such as the role of Accessibility Chair for CHI Play 2018 and 2019, and been a student volunteer at multiple conferences that I have attended during my PhD.
As an active long standing member of the HCI group at The University of Melbourne, I am knowledgeable in a wide range of topics in HCI, and have past and ongoing internal and external research collaborations.
Wang, C., Jiang, W., Yang, K., Newn, J., Sarsenbayeva, Z., Goncalves, J. & Kostakos, V. (2021). A Systematic Review of Electronic Monitoring Systems for Hand Hygiene. In Journal of Medical Internet Research (JMIR). 23, 11. [DOI]
Khan, AA., Newn, J., Kelly, RM., Srivastava, N., Bailey, J. & Velloso, E. (2021). GAVIN: Gaze-Assisted Voice-based Implicit Note-taking. In ACM Transactions on Computer-Human Interaction (TOCHI). 28, 4, Article 26 (July 2021). [DOI]
Yu, D., Zhou, Q., Newn, J., Dingler, T., Velloso, E. & Goncalves, J. (2020). Fully-Occluded Target Selection in Virtual Reality. In IEEE Transactions on Visualization and Computer Graphics (TVCG). 26, 12, pp. 3402-3413 (December 2020). [DOI] [Video]
— Best Paper Nomination (ISMAR 2020)
Zhou, Q., Yu, D., Reinoso, M., Newn, J., Goncalves, J. & Velloso, E. (2020). Eyes-free Target Acquisition During Walking in Immersive Mixed Reality. In IEEE Transactions on Visualization and Computer Graphics (TVCG). 26, 12, 3423-3433 (December 2020). [DOI]
Singh, R., Miller, T., Newn, J., Velloso, E., Vetere, F. & Sonenberg, L. (2020). Combining Gaze and AI Planning for Online Human Intention Recognition. In Artificial Intelligence. 284, Article 103275 (July 2020). [DOI]
Abdelrahman, Y., Khan, AA., Newn, J., Velloso, E., Safwat, S., Bailey, J., Bulling, A., Vetere, F. & Schmidt, A. (2019). Classifying Attention Types with Thermal Imaging and Eye Tracking. In Proceedings of ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). 3, 3, Article 69 (September 2019). [DOI]
Srivastava, N., Newn, J. & Velloso, E. (2018). Combining Low and Mid-Level Gaze Features for Desktop Activity Recognition. In Proceedings of ACM Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). 2, 4, Article 189 (December 2018). [DOI]
Velloso, E., Carter, M., Newn, J., Esteves, A., Clarke, C. & Gellersen, H. (2017). Motion Correlation: Selecting Objects by Matching Their Movement. In ACM Transactions on Computer-Human Interaction (TOCHI). 24, 3, Article 22 (April 2017). [DOI]
— TOCHI Best Paper Award 2017
Khan, AA., Newn, J., Bailey, J., & Velloso, E. (Forthcoming). Integrating Gaze and Speech for Enabling Implicit Interactions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). ACM, New York, NY, USA. [DOI]
Khan, AA., Nawas, S., Newn, J., Kelly, RM., Lodge, JM., Bailey, J., & Velloso, E. (Forthcoming). To Type or To Speak? The Effect of Input Modality on Text Understanding During Note-taking. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). ACM, New York, NY, USA.
Crocher, V., Singh, R., Newn, J., & Oetomo, J. (2021). Towards a Gaze-Informed Movement Intention Model for Robot-Assisted Upper-Limb Rehabilitation. In Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC'21).
Rogerson, MJ., Newn, J., Singh, R., Baillie, E., Benke, L., Papasimeon, M. & Miller, T. (2021). Observing Multiplayer Boardgame Play at a Distance . In Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play. (CHI PLAY '21). ACM, 262-267. [DOI]
Srivastava, N., Nawaz, S., Newn, J., Lodge, J., Velloso, E., Erfani, SM., Gasevic, D., & Bailey, J. (2021). Are you with me? Measurement of Learners' Video-Watching Attention with Eye Tracking. In LAK21: 11th International Learning Analytics and Knowledge Conference (LAK21). ACM, New York, NY, USA, 88–98. [DOI]
Babaei, E., Srivastava, N., Newn, J., Zhou, Q., Dingler, T., & Velloso, E. (2020). Faces of Focus: A Study on the Facial Cues of Attentional States. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, NY, USA. [DOI]
Newn J., Singh, R., Velloso, E., & Vetere, F. (2019). Combining Implicit Gaze and AI for Real-Time Intention Projection. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers (UbiComp '19 Adjunct). [DOI]
Newn, J., Singh, R., Allison, F., Madumal, P., Velloso, E., & Vetere, F. (2019). Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents. In 17th IFIP International Conference on Human-Computer Interaction — INTERACT 2019 (INTERACT '19). [DOI]
Wouters, N., Kelly, RM., Velloso, E., Wolf, K., Ferdous, HS., Newn, J., Joukhadar, Z., & Vetere, F. (2019). Biometric Mirror: Exploring Ethical Opinions towards Facial Analysis and Automated Decision-Making. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS '19). ACM, New York, NY, USA, 447-461. [DOI]
Zhou, Q., Newn, J., Srivastava, N., Goncalves, J., Dingler, T., & Velloso, E. (2019). Cognitive Aid: Task Assistance Based on Mental Workload Estimation. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, New York, NY, USA, Paper LBW2315, 6 pages. [DOI]
Allison, F., Newn, J., Smith, W., Gibbs, M., & Carter, M. (2019). Frame Analysis of Voice Interaction in Gameplay. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Paper 393, 14 pages. [DOI]
Singh, R., Miller, T., Newn, J., Sonenberg, L., Velloso, E., & Vetere, F. (2018). Combining Planning with Gaze for Online Human Intention Recognition. In Proceedings of the 17th Conference on Autonomous Agents and MultiAgent Systems (AAMAS '18). Richland, SC. [URL]
Newn, J., Allison, F., Velloso, E., & Vetere, F. (2018). Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA. [DOI]
Newn, J. (2018). Enabling Intent Recognition Through Gaze Awareness. In Proceedings of the 2018 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA. [DOI]
Newn, J., Velloso, E., Allison, F., Abdelrahman, Y. & Vetere, F. (2017). Evaluating Real-Time Gaze Representations to Infer Intention in Competitive Turn-Based Strategy Games. In Proceedings of the 2017 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17). ACM, New York, NY, USA, 541–552. [DOI]
Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 251–260. [DOI] [PDF]
Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Exploring the Effects of Gaze Awareness on Multiplayer Gameplay . In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (CHI PLAY Companion '16). ACM, New York, NY, USA, 239–244. [DOI] [PDF] [Poster]
Carter, M., Newn, J., Velloso, E., & Vetere, F. (2015). Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '15). ACM, New York, NY, USA, 167–176. [DOI]
Newn, J. (2020). Using Multimodal Sensing to Improve Awareness in Human-AI Interaction. In CHI 2020 Workshop on Artificial Intelligence for HCI: A Modern Approach. Honolulu, USA.
Newn J., Tag, B., Singh, R., Velloso, E., & Vetere, F. (2019). AI-Mediated Gaze-Based Intention Recognition for Smart Eyewear: Opportunities & Challenges. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers (UbiComp '19 Adjunct). London, United Kingdom. [DOI]
Zhou, Q., Newn J., Tag, B., Lee, HP., Wang, C., & Velloso, E. (2019). Ubiquitous Smart Eyewear Interactions using Implicit Sensing and Unobtrusive Information Output. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers (UbiComp '19 Adjunct). London, United Kingdom. [DOI]
Newn, J., Singh, R., Allison, F., Madumal, P., Velloso, E. & Vetere, F. (2019). Nonverbal Communication in Human-AI Interaction. In INTERACT '19 Workshop on Human(s) in the Loop – Bringing AI & HCI Together. Paphos, Cyprus. [URL]
Newn, J., Singh, R., Madumal, P., Velloso, E. & Vetere, F. (2018). Designing Explainable AI Interfaces through Interaction Design Techniques. In OzCHI '18 Workshop on Interaction Design for Explainable AI. Melbourne, Australia.
Matthews, O., Sarsenbayeva, Z., Jiang, W., Newn, J., Velloso, E., Clinch, S. & Goncalves, J. (2018). Inferring the Mood of a Community From Their Walking Speed: A Preliminary Study. In Adjunct Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’18 Adjunct). Singapore, Singapore. [DOI]
Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Dynamically Exposing Gaze to Foster Playful Experiences in Multiplayer Gameplay. In CHI PLAY 2016 Workshop on Designing for Emotional Complexity in Games: The Interplay of Positive and Negative Affect. Austin, TX, USA. [PDF]
Ali, AE., Perusquia-Hernandez, M., Hassib, M., Abdelrahman, Y., & Newn, J. (2021). MEEC: Second Workshop on Momentary Emotion Elicitation and Capture. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA '21). ACM. [URL]
Madumal, P., Singh, R., Newn, J., & Vetere, F. (2018). Interaction Design for Explainable AI. In Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI '18). Melbourne, VIC, Australia.
Lankes, M., Newn, J., Maurer, B., Velloso, E., Dechant, M. & Gellersen, H. (2018). EyePlay Revisited: Past, Present and Future Challenges for Eye-Based Interaction in Games. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (CHI PLAY Extended Abstracts '18'). Melbourne, VIC, Australia.
Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents
2019 IFIP International Conference on Human-Computer Interaction (INTERACT '19),
Coral Beach Hotel & Resort, Coral Bay, Peyia, Cyprus.
Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay
2018 CHI Conference on Human Factors in Computing Systems (CHI '18),
Palais des Congrès de Montréal, Montréal, QC, Canada.
Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games
2017 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17),
Pakhuis de Zwijger, Amsterdam, North Holland, Netherlands.
Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze
2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '16),
Crowne Plaza Niagara Falls, Niagara Falls, ON, Canada.
The Effect of Gaze on Gameplay in Co-located Multiplayer Gaming Environments
2016 Computing and Information Systems Doctoral Colloquium,
Melbourne School of Engineering, Parkville, VIC, Australia.
Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback
2015 Annual Meeting of the Australian SIG for Computer Human Interaction (OzCHI '15),
Melbourne School of Design, Parkville, VIC, Australia.
AI-Mediated Gaze-Based Intention Recognition for Smart Eyewear: Opportunities & Challenges
2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers,
Queen Elizabeth II Centre, London, United Kingdom.
Ubiquitous Smart Eyewear Interactions using Implicit Sensing and Unobtrusive Information Output
2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable Computers,
Queen Elizabeth II Centre, London, United Kingdom.
Enabling Intent Recognition Through Gaze Awareness
2018 CHI Conference on Human Factors in Computing Systems (CHI '18) Doctoral Consortium,
Palais des Congrès de Montréal, Montréal, QC, Canada.
Investigating Intent Recognition Through Gaze Awareness in User Interfaces
2017 Computing and Information Systems Doctoral Colloquium,
Sidney Myer Asia Centre, The University of Melbourne, Parkville, VIC, Australia.
— Best Poster Award
Nonverbal Communication in Human-AI Interaction: Utilising Gaze for Intention Recognition to Facilitate Collaborative Decision-Making (PhD Completion)
Interaction Design Group Seminar (31 May 2019),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.
Exploring Gaze Input in Social User Interfaces
hciLab Seminar (25 October 2017),
SimTech Building, The University of Stuttgart, Stuttgart, Germany.
Disentangling Gaze Cues in Intelligent User Interfaces: Opportunities & Challenges (PhD Confirmation)
Interaction Design Group Seminar (24 February 2017),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.
Gaze Patterns in Social Play: Insights into the Mind of Players
Interaction Design Group Seminar (19 August 2016),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.
Exploring the Combination of Eye Gaze and Gesture on Large Displays
Interaction Design Group Seminar (18 September 2015),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.