Joshua Newn

Joshua Newn, PhD Candidate,
The University of Melbourne

Hello! I'm Joshua, a PhD Candidate at the Microsoft Research Centre for Social NUI at The University of Melbourne under the supervision of Professor Frank Vetere and Dr Eduardo Velloso.

My current research aims to advance the use of eye tracking in social contexts to enrich the interaction between humans and intelligent user interfaces. During face-to-face interaction, humans naturally monitor the gaze of one another as they offer a powerful 'window into the mind'. This human ability should exist in Human-AI interaction, allowing systems to make social inferences through our gaze behaviours. I am currently in the third phase of this research, more information can be found here.

I am currently hired as the Usability Lab Manager for the Interaction Design Lab at the university where I support a wide range of HCI research projects and UX evaluations internally and externally.

In 2015, I completed a Master of Information Systems (Research Specialisation) at the university that converges on multimodal interaction, particulary with eye tracking and touch(+less) gestures, on large interactive surfaces under the supervision of Professor Frank Vetere and Dr Marcus Carter. My current research interests are in the role of gaze cues in human-computer interaction, the implicit/explicit use of eye movements for interaction in games and the use of gaze as a complementary input in multimodal interaction.

We have received over 60 sign ups for this study which is more than we anticipated. We would like to thank everyone for their interest and their time. I'm glad to announce that we will have another study in the coming months that also involves eye tracking and games and if you are interested in taking part, please free to email me at joshua.newn@unimelb.edu.au stating your interest and I will add you to the list.

Are you good at reading the next moves of players when playing a boardgame?
If you could see where they were looking at, would you be able to make better inferences? Challenge yourself by taking part in this exciting study!

We are currently assessing the preferences and ability of humans in making inferences of player strategies during competitive gameplay with the help of eye-tracking. This exciting study involves watching clips of actual gameplay that we enhanced with visuals showing the opponent's gaze. For each recording, you will be asked to tell us what the opponent was thinking of at that time. Once all clips have been shown, you will then be asked to provide your preferences on how gaze was displayed and be given the opportunity to provide any feedback. The whole process will take approximately 30 minutes. This study will inform the next generation of eye tracking enhanced games.

What are eye tracking enhanced games?

Eye tracking enhanced games are games that have knowledge of the players eyes, allowing new ways of interaction with the game environment and characters. This gives players a more immersive experience as they are able to mimic actions that occur in real life, such as looking for cover or aiming at a target. In short, any game that input from the eyes in order to enhance the gameplay experience can be classified as 'eye tracking enhanced games'. Such games are playable without the use of an eye tracker. Deus Ex: Mankind Divided, Steep and Watch Dogs 2 are just some of the popular titles that are enhanced with eye tracking.

Fig. 1: Illustration of gaze-enhanced competitive gameplay.

Fig. 2: The red dot shows where your opponent is currently looking, can you predict the next move?
This is an example of what you will be asked to do.

Awards

Google PhD Travel Scholarship
Computing and Information Systems,
The University of Melbourne Melbourne Abroad Travelling Scholarship
Computing and Information Systems,
The University of Melbourne Australian Postgraduate Award
Computing and Information Systems,
The University of Melbourne Dean's Honours List 2015
Computing and Information Systems,
The University of Melbourne SocialNUI Postgraduate Top-Up Scholarship
Microsoft Research Centre for Social Natural User Interfaces,
Computing and Information Systems,
The University of Melbourne

Education

Doctor of Philosophy (Engineering)
Microsoft Research Centre for Social Natural User Interfaces,
Computing and Information Systems,
The University of Melbourne
2016 - Present Master of Information Systems (Research Specialisation)
Microsoft Research Centre for Social Natural User Interfaces,
Computing and Information Systems,
The University of Melbourne
2014 - 2015 Bachelor of Information Technology and Systems
Faculty of Information Technology,
Monash University
2012 - 2012

Journals

Velloso, E., Carter, M., Newn, J., Esteves, A., Clarke, C. & Gellersen, H. (2017). Motion Correlation: Selecting Objects by Matching Their Movement. In ACM Transactions on Computer-Human Interaction (TOCHI). 24, 3, Article 22 (April 2017). [DOI]

Conference Papers

Singh, R., Miller, T., Newn, J., Sonenberg, L., Velloso, E., & Vetere, F. (Forthcoming). Combining Planning with Gaze for Online Human Intention Recognition. In Proceedings of the 17th Conference on Autonomous Agents and MultiAgent Systems (AAMAS '18).

Newn, J., Allison, F., Velloso, E., & Vetere, F. (2018). Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA. [DOI]

Newn, J. (2018). Enabling Intent Recognition Through Gaze Awareness. In Proceedings of the 2018 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA. [DOI]

Newn, J., Velloso, E., Allison, F., Abdelrahman, Y. & Vetere, F. (2017). Evaluating Real-Time Gaze Representations to Infer Intention in Competitive Turn-Based Strategy Games. In Proceedings of the 2017 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17). ACM, New York, NY, USA, 541–552. [DOI]

Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 251–260. [DOI] [PDF]

Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Exploring the Effects of Gaze Awareness on Multiplayer Gameplay. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (CHI PLAY Companion '16). ACM, New York, NY, USA, 239–244. [DOI] [PDF] [Poster]

Carter, M., Newn, J., Velloso, E., & Vetere, F. (2015). Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '15). ACM, New York, NY, USA, 167–176. [DOI]

Workshop papers

Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Dynamically Exposing Gaze to Foster Playful Experiences in Multiplayer Gameplay. In CHI PLAY 2016 Workshop on Designing for Emotional Complexity in Games: The Interplay of Positive and Negative Affect. Austin, TX, USA. [PDF]

Presentations

Paper Presentations

Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay
2018 CHI Conference on Human Factors in Computing Systems (CHI '18),
Palais des Congrès de Montréal, Montréal, QC, Canada.

Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games
2017 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17),
Pakhuis de Zwijger, Amsterdam, North Holland, Netherlands.

Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze
2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '16),
Crowne Plaza Niagara Falls, Niagara Falls, ON, Canada.

The Effect of Gaze on Gameplay in Co-located Multiplayer Gaming Environments
2016 Computing and Information Systems Doctoral Colloquium,
Melbourne School of Engineering, Parkville, VIC, Australia.

Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback
2015 Annual Meeting of the Australian SIG for Computer Human Interaction (OzCHI '15),
Melbourne School of Design, Parkville, VIC, Australia.

Posters

Investigating Intent Recognition Through Gaze Awareness in User Interfaces
Best Poster Award
2017 Computing and Information Systems Doctoral Colloquium,
Sidney Myer Asia Centre, The University of Melbourne, Parkville, VIC, Australia.

Seminar Presentations

Exploring Gaze Input in Social User Interfaces
hciLab Seminar (25 October 2017),
SimTech Building, The University of Stuttgart, Stuttgart, Germany.

Disentangling Gaze Cues in Intelligent User Interfaces: Opportunities & Challenges
Interaction Design Group Seminar (24 February 2017),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.

Gaze Patterns in Social Play: Insights into the Mind of Players
Interaction Design Group Seminar (19 August 2016),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.

Exploring the Combination of Eye Gaze and Gesture on Large Displays
Interaction Design Group Seminar (18 September 2015),
Interaction Design Lab, The University of Melbourne, Parkville, VIC, Australia.