Qiao (Georgie) Jin 金乔

Research

Publications

Browse papers and preprints by year, or search by title, author, or venue.

Google Scholar · Lab publications

2026

Related lab project showing an instructor in a virtual classroom

European Conference on Technology-Enhanced Learning (EC-TEL 2026)

High and Low-Performing Groups in Collaborative VR Video-Based Learning: An Exploratory Study of Group Interaction and Experience

Lingkan Wang, Qianru Lyu, Yu Liu, Qiao Jin

Abstract

Virtual reality (VR) offers immersive opportunities for collaborative learning, yet little is known about how high- and low-performing groups differ in their collaborative processes and learning experiences. In this study, 54 students (18 groups) participated in a virtual field trip, collaboratively viewing a 360 \(^{\circ }\) video and creating a visitor map in VR. Group outcomes were assessed based on visitor map quality. We then selected the three highest- and three lowest-performing groups and compared their forms of interaction, note-taking behaviors, viewing patterns, and self-reported collaboration, satisfaction, and social presence. Interview data were used to contextualize these findings. Our results showed that high-performing groups displayed declaratory comments paired with more questioning and evaluative discourse, while low-performing groups focused on surface-level content sharing and activity planning without deeper elaboration. High-performing groups also tended to show lower view similarity and a preference for text-based note-taking. Interestingly, descriptive questionnaire results indicated that low-performing groups reported higher satisfaction, social presence, and collaborative experience than high-performing groups. Qualitative findings further contextualize these patterns and inform pedagogical and design implications for collaborative video-based learning in VR.

Related lab project on mixed-reality teacher orchestration

European Conference on Technology-Enhanced Learning (EC-TEL 2026)

“Maybe They Needed Help, but They Weren’t Asking for It”: Teachers’ Decision Making in AI-Supported Mixed Reality Classroom Orchestration

Qianru Lyu, Víctor Alonso-Prieto, Qiao Georgie Jin, Will Morgus, Kyle Price, Stephen Fancsali, Susan R. Berman, Steve Ritter, Vincent Aleven

Abstract

Prior research shows that a Mixed Reality (MR) teacher awareness tool, used by teachers when students practice with AI-based tutoring software, can shift teachers’ attention to students who need help the most and can help students learn better. However, how teachers make sense of, prioritize, and act on MR-provided analytics during live instruction remains underexplored, limiting the pedagogical potential of real-time analytics. The current study examines teachers’ noticing and instructional agency in this context. We compare teacher classroom orchestration with and without MR support and investigate how teachers notice, interpret, and act on MR-provided information in real time. Semi-structured interviews were conducted with four middle school mathematics teachers who used the MR tool in their classrooms. Based on the interview data, qualitative coding and Epistemic Network Analysis (ENA) were used to model patterns between teacher noticing, interpretation, and instructional actions. Teachers reported that compared to normal classrooms, the MR tool helped them shift toward more continuous awareness, need-based prioritization, and more calibrated intervention practices. Distinct decision-making trajectories emerged, in which recurrent patterns of quick nudges, sustained instruction, or deliberately delayed intervention were conditioned on both signals of idling and struggling, communicated by the tools and teachers’ professional knowledge of students and tasks. The findings provide a nuanced account of teachers’ decision-making process in MR-supported classroom orchestration. The insights about teachers’ decision-making process may also contribute to the development of a recommendation system that suggests who to help, as well as the design of teachers’ professional development for everyday AI-supported classrooms.

IEEE Transactions on Visualization and Computer Graphics (TVCG) · 2026

Signals of AI Hallucination: Designing Hallucination-Aware Cues for Embodied Conversational Agents in VR

Xiaoran Yang, Yang Zhan, Xie He, Yuxuan Huang, Yichen Yu, Zhuo Wang, Noboru Matsuda, Qiao Jin

Abstract

LLM-powered conversational agents (CAs) often present uncertainty and provenance cues alongside their responses to help users assess response reliability and identify potential hallucinations. In immersive environments such as Virtual Reality (VR), CAs often take the form of speech-based embodied conversational agents (ECAs), where uncertainty and provenance cues cannot rely on persistent inline text and may be missed or disrupt comprehension when delivered through speech. We conducted a within-subjects study (N = 24) to compare three designs for presenting the hallucination-awareness information (uncertainty and provenance) in ECAs in VR against a no-cue baseline: embodied cues using gestures and posture, icon cues using visual indicators, and text cues using color-coded text with inline citations. We evaluated how these designs affect users' ability to identify hallucination-related information, trust in the ECA, and interaction experience (immersion and task load). Our results show that all three designs support users in identifying hallucinations. Embodied cues were associated with higher trust and immersion, text cues offered clearer interpretability, and icon cues preserved relatively good interpretability while causing less disruption to immersion compared with embodied cues and text cues. This work contributes to the VR and AI research community by comparing different designs of hallucination cues in immersive ECA settings and examining how they affect users' ability and experiences to identify hallucinations. It also offers practical insights and design implications for developing future hallucination-awareness interfaces for ECA.

ACM Symposium on Virtual Reality Software and Technology (VRST 2026)

Context When Needed, Digits When Useful: Balancing Memorability and Efficiency in Virtual Reality Authentication

Yuxuan Huang, Qiao Jin, Tongyu Nie, Victoria Interrante, Evan Suma Rosenberg

Abstract

We present Adaptive Direction-Based Authentication (ADBA), a knowledge-based authentication method for Virtual Reality that decouples users' needs temporally by enforcing password creation based on virtual environment context while supporting both context- and digit-based entries during authentication. This design prioritizes memorability for new passwords and offers both efficient and memorable options to support users' evolving needs. We conducted a remote longitudinal study with 66 participants comparing ADBA against 6-digit PINs over 2-3 weeks. The results demonstrated that ADBA achieved superior memorability and lower perceived task load. Interestingly, no participant chose to enter via digits in the study, yet they still perceived ADBA to be highly efficiency despite longer objective entry times. ADBA also provided security benefits through randomly-generated digit representations, though some degree of password homogeneity was observed in specific virtual environments. Our findings suggest that ADBA offers solid advantages to the traditional PINs, and successfully addresses the tradeoffs between efficiency, memorability, and security under the usage scenarios considered in the study.

ACM Transactions on Applied Perception (TAP) · 2026

How Does Gamification Affect Virtual Reality Nature Experience?

Yuxuan Huang, Qiao Jin, Sara Lotemplio, Francisco R. Ortega, Victoria Interrante

Abstract

Studies have shown that simulated nature in virtual reality (VR) can achieve restorative effects similar to real nature. To avert boredom and enhance existing virtual nature experience, researchers are exploring the potential of incorporating interactive and gamified elements to promote engagement and effectiveness. While relaxation games have restorative potential, prior studies did not consider how a gamified objective can affect mental load and whether it would interfere with “soft fascination”, which is the key mechanism of nature restoration according to the attention restoration theory. This study investigated the impact of including a gamified objective on VR nature experiences with respect to both psychological and cognitive restorative outcomes and user preference. We conducted a two-session within-subjects experiment comparing two conditions with similar nature environment and identical interaction features. In the hard fascination condition (HF), participants were instructed to find and take pictures of target objects; in the soft fascination condition (SF) they were instructed to freely explore and take pictures of whatever they liked. The quantitative results showed that incorporating goal-directed gamified tasks in VR nature experience could enhance cognitive performance but reduce relaxation benefits. The qualitative findings from semi-structured interviews triangulated the quantitative results and revealed additional benefits of gamification, such as directing user attention and promoting long-term engagement. The findings suggest that gamifying a VR nature experience introduces a tradeoff between relaxation and other benefits, such as engagement and cognitive performance. As such, gamification may be most appropriate in contexts where these alternative benefits are prioritized over relaxation alone.

First three stages of the computational design partner framework: express intent, generate artifacts, and simulate behavior.

Next-Generation Learning Interfaces Workshop at the International Conference on Artificial Intelligence in Education (AIED 2026)

From Code Generation to Computational Design: Generative AI as an Interactive Partner for Future Computer Science Education

Yichen Andy Yu, Qiao Jin

Abstract

Recent advances in generative AI have made it easier for students to generate code, debug errors, and receive programming explanations through natural language prompts. However, code generation alone does not ensure that students understand the computational reasoning behind a program. Current AI coding tools may help students produce syntactically correct solutions while bypassing key learning processes such as problem decomposition, abstraction, algorithmic reasoning, debugging, and evaluation. In this position paper, we argue that generative AI for computer science education should move beyond answer generation and become a computational design partner. Such systems should help learners express computational intent, transform that intent into inspectable intermediate representations, simulate program behavior, and support iterative refinement. We propose a framework in which students express intent, AI generates plans or scaffolds, students inspect and revise behavior, and the system explains the consequences of their changes. Our goal is to support students in learning how to design, reason about, and evaluate computational systems rather than simply obtain final code.

International Conference on Artificial Intelligence in Education (AIED 2026)

Designing and Evaluating Next-Generation Learning Interfaces: Linking AI, HCI, and the Learning Sciences

Meng Xia, Yan Chen, Qiao Jin, Yang Shi, Paul Denny, Tiffany Barnes, Qingsong Wen, Vincent Aleven

Abstract

We propose to host the full-day workshop intended for the AIED 2026 conference. Education is being reshaped by rapid advances in generative AI and immersive technologies, which offer unprecedented opportunities for scalable tutoring, personalized feedback, multimodal engagement, and new forms of human–AI collaboration, while also raising challenges such as over-reliance on AI-generated content, erosion of learner agency, issues of trust and bias, and accessibility in immersive environments. Addressing these opportunities and challenges is inherently interdisciplinary, requiring expertise from AI, Human-Computer Interaction (HCI), and learning sciences to ensure that emerging systems are both effective and responsible. This 2026 learning festival workshop brings together researchers from AI, HCI, and learning sciences communities to chart the future of next-generation learning interfaces. Through lightning talks, paper presentations, and collaborative activities, participants will share methods, examine the roles of humans and AI in complex learning scenarios, and identify research agendas that bridge theory and practice, with outcomes including a shared knowledge base of design strategies grounded in learning theory, a roadmap for evaluating emerging educational technologies, and a lasting interdisciplinary community at the intersection of AI, HCI, and education.

International Conference on Artificial Intelligence in Education (AIED 2026)

Using an MR-Based Teacher Orchestration Tool in AI-Supported K-12 Classrooms

Qiao Jin, Yang Zhan, Will Morgus, Kyle Price, Michael Sandbothe, Jonathan Sewall, Octav Popescu, Susan Berman, Stephen E. Fancsali, Steve Ritter, Kenneth Holstein, Vincent Aleven

Abstract

Intelligent tutoring systems (ITSs) can personalize learning in K-12 classrooms, while teachers still need to make real-time decisions about how to allocate support across students. Mixed Reality (MR) teacher tools have shown promise for supporting this work, yet evidence from authentic classrooms remains limited. We report an in-the-wild field study of Lumilo2, an MR-based teacher orchestration tool integrated with MATHia and deployed in middle school mathematics classrooms. Across seven classroom sessions involving six teachers and 113 students, we examined how the system was used in practice and perceived by teachers and students. Teachers valued the tool’s rapid, hands-free support for noticing student needs, while also wanting more specific, prioritized, and configurable information. Students were generally positive about their teacher’s use of the tool, although some raised privacy concerns.

ACM Interaction Design and Children Conference (IDC 2026)

Child-Involved Group Prioritization: Effects of Group Composition on Experience, Participation Level, and Interaction

Qiao Jin, Leah Ajmani, Samantha Singh, Adhitya Balasubramanian, Svetlana Yarosh

Abstract

Stakeholder-informed prioritization (SIP) is a lightweight technique designed to help child-involved groups quickly understand and prioritize research and design directions. However, outcomes of short prioritization discussions can vary with group composition. In this paper, we deployed SIP with 34 groups of children and parents (N = 94) at a large community event, focusing on how group composition, including group type (family-based vs. child-based) and familiarity (mixed-familiarity vs. fully familiar), shapes SIP experiences. Results suggest that child-based groups were associated with higher satisfaction, more active child participation, smaller in-group range in comfort, satisfaction, and participation level, and more verbal reasoning. Fully familiar groups reported higher comfort and showed more verbal reasoning and less facilitator involvement, while mixed-familiarity groups showed a smaller in-group range in participation levels. We discuss implications for designing and facilitating child-involved group prioritization sessions and provide practical guidance for using SIP as a rapid method in early-stage research.

Companion Publication of the 2026 ACM Designing Interactive Systems Conference (DIS Companion 2026)

SpatialPrompt: XR-Based Spatial Intent Expression as Executable Constraints for AI Generative 3D Design

Yichen Andy Yu, Wanru Li, Qiaoran Wang, Jymon Ross, Gavin Johnson, Mandy Lui, Qiao Jin

Abstract

We present SpatialPrompt, an Extended Reality(XR) system that turns spatial sketches into executable constraints for controllable 3D generation. Users draw rough structures with a 3D pen and add voice prompts for semantic and stylistic intent. The system supports iterative refinement and synchronous co-creation in shared space with color-coded contributions. Implemented on Apple Vision Pro with Logitech Muse and Meshy, a heuristic evaluation suggests that the workflow is intuitive and supports shared understanding in collaborative creation, while revealing needs for faster generation and clearer feedback.

Companion of the Computer-Supported Cooperative Work and Social Computing (CSCW Companion 2026)

Growing Up (and Old) with AI: Co-Constructing the Future for Family-Centered AI

Xiaoyi Tian, Renkai Ma, Qiao Jin, Kaiwen Sun, Afsaneh Razi, Yang Shi, Bengisu Cagiltay, Jerry Alan Fails, Yubo Kou

Abstract

As AI tools become more accessible to the general public, their presence and prominence within family contexts are increasing. However, similar to existing interactive technologies, AI tools are rarely tailored for collective use within an interconnected family unit. This overlooks the opportunities for shared AI use within families, as well as the conflicts and negotiations necessary to manage it. Bridging expertise across AI in education, child-computer interaction, social computing, and family studies within the CSCW community, this workshop convenes researchers to examine how AI shapes intergenerational dynamics, from child group activities, we will co-construct a research and design agenda to build an interdisciplinary, collaborative community of researchers and practitioners dedicated to designing an equitable, value-sensitive future for families navigating life alongside AI.

International Conference on Learning Analytics and Knowledge (LAK 2026)

Sticky Help, Bounded Effects: Session-by-Session Analytics of Teacher Interventions in K-12 Classrooms

Qiao Jin, Conrad Borchers, Ashish Gurung, Sean Jackson, Sameeksha Agarwal, Cancan Wang, YiChen Yu, Pragati Maheshwary, Vincent Aleven

Abstract

Teachers’ in-the-moment support is a limited resource in technology-supported classrooms, and teachers must decide whom to help and when during ongoing student work. However, less is known about how students’ prior help history (whether they were helped earlier) and their engagement states (e.g., idle, struggle) shape teachers’ decisions, and whether observed learning benefits associated with teacher help extend beyond the current class session. To address these questions, we first conducted interviews with nine K–12 mathematics teachers to identify candidate decision factors for teacher help. We then analyzed 1.4 million student–system interactions from 339 students across 14 classes in the MATHia intelligent tutoring system by linking teacher-logged help events with fine-grained engagement states. Mixed-effects models show that students who received help earlier were more likely to receive additional help later, even after accounting for current engagement state. Cross-lagged panel analyses further show that teacher help recurred across sessions, whereas idle behavior did not receive sustained attention over time. Finally, help coincided with immediate learning within sessions, but did not predict skill acquisition in later sessions, as estimated by additive factor modeling. These findings suggest that teacher help is “sticky” in that it recurs for previously supported students, while its measurable learning benefits in our data are largely session-bound. We discuss implications for designing real-time analytics that track attention coverage and highlight under-visited students to support a more equitable and effective allocation of teacher attention.

Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI 2026)

When AI Gets It Wrong: Scaffolding AI Hallucination Detection for Children Through Chatbot Creation

Xiaoyi Tian, Deniz Ozturk, Sreekar Edula, Jibran Adil, Qiao Jin, Yang Shi, Tiffany Barnes

Abstract

Children increasingly interact with generative AI systems that can produce hallucinated content, potentially reinforcing misconceptions and undermining critical thinking skills. We investigate how children detect and respond to hallucinations while building and testing LLM-powered chatbots in a development environment. We integrated hallucination-awareness scaffolds such as confidence indicators, fact-checking, repeated questioning, and model comparison. Through a study with 48 middle school learners aged 10-14, participants showed significant pre-to-post gains in AI knowledge, hallucination awareness, and confidence in building trustworthy chatbots. They developed multi-layered strategies, including probing inconsistencies and cross-checking with external sources. Key challenges included over-reliance on visible cues, fragmented use of scaffolds, and a tension between creativity and reliability. These findings highlight design implications for children’s AI literacy for responsible AI development: supporting proactive, iterative engagement in the development cycle, integrating scaffolds into coherent workflows, and balancing creativity with accuracy.

Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA 2026)

Understanding Prompt Construction and Interaction Challenges in LLM-Based Multi-Robot Systems for Spatial Tasks

Xiaoran Yang, Yang Zhan, Qiao Jin

Abstract

LLM-based multi-robot systems (MRS) can take a user’s high-level goal and generate coordinated robot actions for spatial tasks. However, human-in-the-loop is still needed when the system benefits from humans providing clarifications or preferences, especially in settings where full autonomy is not yet reliable and the environment is highly uncertain. Motivated by this gap, we conducted a qualitative study (N = 12) in a VR-based multi-robot simulation environment to examine how users construct prompts for MRS and what challenges they perceive during task decomposition, allocation, and execution. Results show that users use prompts to specify robot roles, order, and end goals, and re-prompt when results are inefficient, or execution is not observable. We also found that gaps in workflow visibility, interface support, and coordination logic hinder users’ understanding and their ability to intervene. We provide design implications for human-in-the-loop interfaces and prompt support in LLM-based MRS.

Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA 2026)

Playground+: an AI-MR System for Adaptable Open-Ended Family Physical Activity Play

Hongfei Ji, Qiao Jin

Abstract

Physical activity play (PAP) is vital for children’s physical, cognitive, and social development, and previous systems and research have explored how various technologies can support it. However, existing approaches often offer limited game choices, rely heavily on family initiative for ideation, and do not align well with everyday, open-ended play practices. AI-augmented mixed reality (AI-MR) systems offer new opportunities through improved contextual awareness, support for play ideation that lowers barriers for families, and wearable-free visual guidance. We present Playground+, a projection-based AI–MR system that combines computer vision and large language models to ideate, generate, and guide PAP. Using a ceiling-mounted projector and webcam, the system delivers multimodal guidance that accommodates different parental roles and children’s developmental levels. Future work will focus on designing temporal and social support, exploring different possibilities of open-ended play, comparing with spatially flexible solutions through user studies, and addressing technical and deployment challenges.

IEEE Transactions on Visualization and Computer Graphics (TVCG) · 2026

“We Care More About What It Does”: Priorities and Responsibilities in VR for Children and Families

Qiao Jin, Xiaoran Yang, Svetlana Yarosh

Abstract

Families make key decisions about whether and how Virtual Reality (VR) enters children’s lives, but their priorities rarely shape the research agendas and accountability structures that guide what is studied and built. We examine children’s and families’ views on (1) which knowledge is most needed to support decisions about children’s home VR use, (2) which advances, tools, and content should be prioritized, and (3) which actors they see as responsible for addressing these priorities. We first conducted a secondary analysis of prior interviews with children and guardians to distill a structured set of family-perceived alternatives for home VR use. We then engaged 46 child-involved groups (84 children, 40 guardians) to collaboratively prioritize the perceived importance of these alternatives and assign responsibility to different actors. We found that families prioritized knowledge about VR’s effects on brain development and behavior as the most needed, viewing it as the underlying mechanisms with downstream consequences for other effects. They treated safety monitoring as a prerequisite that should not depend solely on parental control, and ranked safety above affordability, comfort, and visual realism. This work contributes to the VR community by offering a childrenand family-informed VR research agenda, along with insights into who families believe should be responsible for addressing these priorities.

arXiv · 2026

Secure Text Entry using a Virtual Radial Keyboard with Dynamically Resized Keys and Non-Intrusive Randomization

Yuxuan Huang, Qiao Jin, Tongyu Nie, Victoria Interrante, Evan Suma Rosenberg

Abstract

As virtual reality (VR) becomes more widely adopted, secure and efficient text entry is an increasingly critical need. In this paper, we identify a vulnerability in a state-of-the-art secure VR text entry method and introduce a novel virtual radial keyboard designed to achieve a balance between security with usability. Keys are arranged alphabetically in a circular layout, with each key selected by controller rotation and dynamically expanding to facilitate precise selection. A randomized rotation mechanism shifts the keyboard after each keystroke, preserving relative key positions while disrupting absolute spatial mappings to protect against inference attacks. We conducted a within-subject study (N=30) comparing our method with the prior secure technique and a standard QWERTY keyboard. Results showed that the radial keyboard significantly improves resistance to keystroke prediction attacks while incurring a tradeoff in entry speed and subjective workload due to the unfamiliar non-QWERTY layout. However, both quantitative trends and qualitative feedback indicate strong potential for performance improvements with practice. We also discuss design implications, possible interface refinements, and directions for future work, including layout variations and visual enhancements.

Companion of the Computer-Supported Cooperative Work and Social Computing (CSCW Companion 2026)

CollabPEAS: A Collaborative VR System for Learning AI Agent Design Through the PEAS Framework

Wanru Jovie Li, Yichen Andy Yu, Qiao Jin

Abstract

We present CollabPEAS, a collaborative system for artificial intelligence (AI) literacy in which pairs of learners come to understand how an AI agent’s behavior is produced by human specification. Working in shared virtual reality (VR), learners jointly specify a virtual cleaning robot using the PEAS framework (Performance measure, Environment, Actuators, Sensors), developing competencies for reasoning about how AI agents sense, decide, and act. The system integrates three coordinated design elements: (1) a fivestep PEAS-guided design flow, implemented in VR, that makes agent specification choices spatially grounded and observable, (2) a block-based coding interface that preserves the perceive, decide, act reasoning cycle, and (3) collaboration supports designed to structure negotiation and shared ownership between partners. We situate CollabPEAS within AI literacy and collaborative learning research and discuss directions for future evaluation.

2025

IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2025)

GenLARP: Enabling Immersive Live Action Role-Play through LLM-Generated Worlds and Characters

Yichen Yu, Yifan Jiang, Mandy Lui, Qiao Jin

Abstract

We introduce GENLARP, a virtual reality (VR) system that transforms personalized stories into immersive live action role-playing (LARP) experiences. GenLARP enables users to act as both creators and players, allowing them to design characters based on their descriptions and live in the story world. Generative AI and agents powered by Large Language Models (LLMs) enrich these experiences.

IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2025)

From Tool to Partner: Exploring the Roles of Embodiment on AI Agent in Pair Programming

Xiaoran Yang, Yang Zhan, Noboru Matsuda, Qiao Jin

Best Short Paper Honorable Mention

Abstract

Pair programming with AI often faces challenges in productive communication and engagement. Integrating embodiment offers a promising solution by making AI a more engaging and context-aware programming partner. To explore how embodied AI agent supports programming learning and affects user experiences, we designed a virtual reality (VR) programming environment with Wizard-of-Oz-controlled AI agents. Our study collected data from 18 participants through knowledge acquisition assessments and interviews. The results showed that embodiment improved engagement, enhanced communication efficiency, and offered emotional support. Specifically, the incorporation of embodied actions allows users to perceive the AI agent as a programming partner and introduces many interactions that resemble those shared with real-life partners.

International Conference on Educational Data Mining (EDM 2025)

Who to Help? A Time-Slice Analysis of K-12 Teachers' Decisions in Classes with AI-Supported Tutoring

Qiao Jin, Conrad Borchers, Stephen Fancsali, Vincent Aleven

Abstract

Classroom orchestration tools allow teachers to identify student needs and provide timely support. These tools provide real-time learning analytics, but teachers must decide how to respond under time constraints and competing demands. This study examines the relationship between indicators of student states (e.g., idle, struggle, system misuse) and teachers' decisions about whom to help, using data from 15 classrooms over an entire school year (including 1.6 million student actions). We found that teachers primarily helped students based on idleness, while student help seeking in the tutoring system was primarily related to struggle. Furthermore, our findings show that students' receipt of help from teachers was significantly positively correlated with their in-system learning rate.

Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY Companion 2025)

Jump, Stop, Jump Again: Exploring AI-Supported Physical Activity Play at Home with Parents and Children

Hongfei Ji, Yuhan Yuan, Qiao Jin

Abstract

Physical activity play (PAP) is essential for children's physical, cognitive, and social development. However, constraints such as limited space, parental availability, and social and physical incidents make domestic PAP a substitute for outdoor play. Given the varied forms of AI, the diverse stages of child development, and the complexity of parent-child dynamics, it is critical to understand how AI influences PAP in the domestic context. This research investigates how children engage with AI for PAP at home and how parents participate in these interactions. Through a small-scale pilot study with three families, we examine the forms of AI-supported PAP, parental roles, challenges to children's agency in relation to parents and AI, and limitations of current AI systems.

ACM Interaction Design and Children Conference (IDC 2025)

Stakeholder-Informed Prioritization (SIP): A Technique for Quickly Gauging Research and Design Priorities

Qiao Jin, Leah Ajmani, Samantha Singh, Adhitya Balasubramanian, Svetlana Yarosh

Abstract

As technology plays a growing role in children's lives, incorporating their perspectives is crucial for shaping future research and design. This paper introduces a lightweight technique, called Stakeholder-Informed Prioritization (SIP), designed to help child-involved groups quickly get informed about and prioritize research and design directions. We demonstrate the use of SIP through a case study focused on Virtual Reality (VR), a technology that is rapidly gaining popularity among younger users but remains under-explored from their point of view. We piloted SIP with 34 groups of children (aged 8 to 18) and their families during a community event. Our results show that participants in these groups can successfully complete SIPs and are generally satisfied with the results.

ACM International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing (MobiHoc 2025)

Exploring Privacy Challenges in Using Volumetric Video for Educational VR

Yu Liu, Qiao Jin, Feng Qian

Abstract

Volumetric video (VV) offers photorealistic 3D capture for immersive educational VR, often created by instructors through live-streamed lessons or prerecorded demonstrations. While enhancing engagement and presence, such instructor-produced content can unintentionally expose sensitive objects, personal information, or biometric identifiers, and may intensify feelings of surveillance. This poster examines these privacy risks in using VV for educational VR and presents a research agenda focused on integrating diminished reality (DR) techniques and real-time 3D scene understanding into VV pipelines to dynamically sanitize environments while balancing realism and privacy.

International Conference of the Learning Sciences (ICLS 2025)

An Exploratory Study of Using Interactive Volumetric Video in VR for Embodied Learning

Qiao Jin, Yu Liu, Yuxuan Huang, Bo Han, Feng Qian, Svetlana Yarosh

Abstract

Volumetric video (VV) has the potential to revolutionize traditional video-based learning (VBL) by offering immersive, 3D content that enhances student engagement and comprehension. However, the limited interactivity of current pre-recorded VV restricts its educational applications. To address this, we developed and evaluated an interactive VV viewing system that enables learners to engage with spatial interactive areas. These areas trigger multimodal outcomes and support both static and continuous interactions tailored to specific educational objectives. Compared to the basic condition (VV without interaction), the interactive VV condition demonstrated higher germane cognitive load and interest in learning but showed no significant differences in short-term memory retention.

International Conference on Artificial Intelligence in Education (AIED 2025)

Lumilo 2: A Scalable Mixed Reality System for Real-Time Teacher Analytics and Intervention

Qiao Jin, Kyle Price, Will Morgus, Michael Sandbothe, Jonathan Sewall, Octav Popescu, Amy Wu, Luke Freimanis, Ravi Oliveira Nedungadi, Pragati Maheshwary, Susan Berman, Stephen Fancsali, Steve Ritter, Vincent Aleven

Abstract

This interactive event presents Lumilo 2, a scalable mixed reality (MR) orchestration tool designed to support real-time teacher intervention during student learning with intelligent tutoring systems (ITSs). Building on previous work that demonstrated significant learning gains with MR teacher orchestration tools, Lumilo 2 reimagines this technology for modern and lightweight MR devices (Xreal Air 2 Ultra). Beyond real-time awareness, Lumilo 2 scales across ITS platforms and classrooms, advancing the goal of making data-driven teacher support broadly accessible in real-world settings.

Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA 2025)

Chameleon: Unobtrusive Substitution of Real-World Obstacles in VR with Risk-Level-Aware Adaptation

Yichen Yu, Qiao Jin

Abstract

In VR environments, free movement in real space enhances immersion but increases the risk of collisions with real-world obstacles. Prior solutions investigated using substitute obstacles with context-related digital objects in VR but often treat all obstacles uniformly without considering their varying levels of risk. This oversight might result in reduced awareness for high-risk obstacles and a missed opportunity to utilize low-risk objects to enhance haptic feedback and interactivity in VR. In this study, we propose Chameleon, a system that classifies real-world obstacles by their varying risk levels and substitutes them with context-related virtual objects in VR. The substitutions are designed to align with the obstacles' real-world risk levels to ensure both safety and immersion. A preliminary heuristic evaluation assessed the usability of using visual textures to implicitly represent obstacle risk levels.

Proceedings of the ACM on Human-Computer Interaction (PACMHCI), CSCW 2025

Assessing Disparities in Hybrid and Online-Only Local Support Communities

Anjali Srivastava, Qiao Jin, Sabirat Rubya, Carrie Kistler, Diana Vuong, Svetlana Yarosh

Abstract

HCI aspires to achieve equitable outcomes in our sociotechnical interventions. Prior work suggests that improved evaluation and reporting of intervention disparities can help achieve more equitable outcomes. We analyzed socioeconomic disparities in hybrid and online-only local support communities by quantitatively operationalizing the access-adoption-adherence-effectiveness (AAAE) framework. The hybrid intervention demonstrated statistically significant socioeconomic status (SES)-based disparities in adoption of community's asynchronous online platform, while the online-only intervention demonstrated statistically significant disparities in access and adherence to synchronous online classes. Our findings suggest that disparities in earlier elements of the framework pipeline may mask later disparities due to survivorship bias, highlighting the importance of comprehensively examining the four elements of the framework.

International Journal of Child-Computer Interaction · 2025

They All Look Mad with Each Other!: Understanding the Needs and Preferences of Children and Parents in AI-Generated Images for Stories

Qiao Jin, Ye Yuan

Abstract

The growing availability of AI-generated images has opened new possibilities for children's storytelling. However, little is known about how children and parents perceive and evaluate AI-generated images in the context of stories. We conducted a study with children and parents to understand their needs and preferences for AI-generated images in stories, examining factors such as emotional expression, diversity, and visual quality.

2024

Companion of the 2024 Computer-Supported Cooperative Work and Social Computing (CSCW Companion 2024)

Towards Responsible Use of Large Multi-modal AI to Analyze Human Social Behaviors

Qingxiao Zheng, Xi Lu, Qiao Jin, Jitesh Jain, Hedda Meadan-Kaplansky, Humphrey Shi, Jinjun Xiong, Yun Huang

Abstract

This SIG is primarily designed for researchers and practitioners interested in exploring the utilization of multimodal artificial intelligence (AI) to understand and analyze human behaviors across various domains. It will particularly benefit professionals in UX design and research, cognitive science, education, psychology, and data science who are pursuing innovative ways to integrate multimodal data with AI to gain deeper behavioral insights.

Proceedings of the ACM on Human-Computer Interaction (PACMHCI), CSCW 2024

Field Trial of a Tablet-based AR System for Intergenerational Connections through Remote Reading

Ye Yuan, Peter Genatempo, Qiao Jin, Svetlana Yarosh

DEI Recognition

Abstract

Prior work has explored various technology designs to support intergenerational communications and connections through remote activities such as reading or play. However, few works have explored these technologies outside the family settings. In this work, we aimed to understand how technology can support social connectedness through remote activities, by investigating the use of a tablet-based AR system among older adult volunteers and students for remote reading. We developed the system based on insights from previous research, deployed the system in the field, and observed the use of the system over six months. With the data collected from the field, we present a rich description on the use of the system and the practices that emerged around its usage in a real-world setting. Our findings highlight the importance of supporting an engaging reading experience and context understanding for social connections with the technology design.

Proceedings of the ACM on Human-Computer Interaction (PACMHCI), CSCW 2024

Designing Collaborative Technology for Intergenerational Social Play over Distance

Ye Yuan, Qiao Jin, Chelsea Mills, Svetlana Yarosh, Carman Neustaedter

Abstract

Collaborative play not only provides entertainment but also nurtures connections and strengthens community ties. In family and intergenerational contexts, collaborative activities and play can engage family members in quality conversations and meaningful connections over distance. To explore participants' preferences, practices, and interaction dynamics when playing remotely, we conducted a design probe study with 15 groups of parents and children from 16 families. Our findings highlight both similarities and notable differences in the use of communication methods, workspaces, and objects between parents and children. Specifically, we observed distinct patterns in gestural and verbal communication and identified specific challenges encountered by children in a simulated remote setting. Our findings also revealed the dynamics of play sessions, particularly when co-located participants are involved, shedding light on the complexities of remote intergenerational communication and play. Our work contributes empirical insights into designing more effective and engaging remote collaborative platforms for families.

The 37th Annual ACM Symposium on User Interface Software and Technology (UIST Adjunct 2024)

SenseBot: Leveraging Embodied Asymmetric Interaction and Social Robotic to Enhance Intergenerational Communication

Hao Jin, Lyumanshan Ye, Qiao Jin

Abstract

Embodied communication, which uses physical cues to convey meaning and emotions, is essential for building social connections. In this work, we introduce SenseBot, an augmented reality (AR)-enabled robotic representative designed to facilitate embodied communication. SenseBot acts as a local agent for remote users, allowing them to engage more interactively and meaningfully with people on-site.

The 37th Annual ACM Symposium on User Interface Software and Technology (UIST Adjunct 2024)

HoloClass: Enhancing VR Classroom with Live Volumetric Video Streaming

Yu Liu, Bo Han, Qiao Jin, Svetlana Yarosh, Zejun Zhang, Feng Qian

Abstract

Virtual Reality (VR) enhances education by creating immersive and engaging learning environments. Volumetric video (VV) further improves VR classrooms by offering realistic, 3D representations of instructors and materials without high development costs. This study introduces HoloClass, a live VV streaming system for VR classrooms. We conducted interviews with 18 students to identify key design needs, resulting in features of HoloClass that support real-time awareness, classroom scalability, and note-taking. Our contributions include empirical insights into designing educational tools with live VV in VR classrooms and the implementation of features that enhance interaction and learning in a virtual setting.

ACM Interaction Design and Children Conference (IDC 2024)

Is Your Family Ready for VR? Ethical Concerns and Considerations in Children's VR Usage

Qiao Jin, Saba Kawas, Stuti Arora, Ye Yuan, Svetlana Yarosh

Best Paper Honorable Mention

Abstract

Virtual Reality (VR) has emerged as a transformative technology to revolutionize education, socialization, and entertainment. However, alongside its promising applications, concerns about potential risks, particularly for young children, have surfaced. We conducted a survey and interview study with 55 parents and 67 children aged 7-13, to identify ethical concerns and design considerations for VR usage in youth populations. This study contributes to the HCI community by providing empirical insights into the ethical concerns surrounding children's VR usage, as viewed from both children's and their parents' perspectives. These insights stem directly from their perceptions of the risks, benefits, and considerations associated with VR across various usage scenarios. Additionally, our research offers valuable design and research implications for the development of responsible VR practices that balance the advantages and potential risks for children and their families.

International Conference on Computer-Supported Collaborative Learning (CSCL 2024)

Interaction Forms of Collaborative VR Video Learning: An Exploratory Study

Qiao Jin, Ishan Sodhi, Angie Chen, Svetlana Yarosh

Abstract

Collaborative and immersive environments both contribute to increasing engagement in video-based learning (VBL). This exploratory study delves into the dynamics of interaction forms within collaborative VR video-based learning and evaluates how shared and individual video control influence these interactions. Through a within-subject study involving 18 groups of three participants (N = 54), we aim to address three research questions: identifying interaction forms in collaborative VR video-based learning, comparing interaction forms between shared and individual video control, and examining the relationship between interaction forms and knowledge acquisition and satisfaction.

Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 2024)

Virtual Reality, Real Pedagogy: A Contextual Inquiry of Instructor Practices with VR Video

Qiao Jin, Yu Liu, Ye Yuan, Bo Han, Feng Qian, Svetlana Yarosh

Abstract

Virtual reality (VR) offers promise in education given its immersive and socially engaging nature, but it can pose challenges for educators when creating VR-specific content. VR videos can function as a new educational tool for VR content creation due to their creation affordability and user-friendliness. However, little empirical research exists on how educators utilize VR videos and associated pedagogy in real classes. Our research employed a contextual inquiry, through in-person interviews and online surveys with 11 instructors to gain actionable insights from envisioned teaching scenarios for VR videos that are informed by actual instructional practices. Our study aims to understand the factors that motivate instructors' adoption of VR videos, identify challenges educators face when incorporating VR videos into instructional units, and examine pedagogical adjustments when integrating VR videos into teaching. Through empirical evidence, we provide design implications for the development of VR-based learning experiences across diverse educational contexts.

Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA 2024)

Methods for Family-Centered Design: Bridging the Gap Between Research and Practice

Bengisu Cagiltay, Zhaoyuan Su, Qiao (Georgie) Jin, Hui-Ru (Irene) Ho, Yuxing Wu, Junnan Yu, Kaiwen Sun, Olivia Richards, Jerry Alan Fails, Jason Yip, Jodi Forlizzi

Abstract

This SIG brings together researchers and practitioners to discuss family-centered design approaches for technology research and practice. As technology becomes deeply embedded in family life, there is a growing need to consider how design processes can better account for the diverse needs, values, and dynamics within families.

2023

Proceedings of the 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems (ImmerCom 2023)

MagicCloth: Protect User Privacy in AR Streaming

Yuming Hu, Mingyu Zhu, Qiao Jin, Feng Qian, Bin Li

Abstract

With the growing number of users, Augmented Reality (AR) privacy issues have aroused researchers' concerns. Our work focuses on the protection of privacy-sensitive objects in indoor settings. We proposed a novel privacy protection mechanism MagicCloth for AR live streaming. When the user covers the privacy-sensitive object with a specially designed cloth, MagicCloth can accurately track it in real time, and replace it with a virtual object based on its shape and size seamlessly. By wisely integrating lightweight 0-1 segmentation, pattern detection, and AR plane tracking techniques, our MagicCloth system achieves real-time performance on Android smartphones.

ACM Interaction Design and Children Conference (IDC 2023)

Exploring Embodied Approaches for Large Age Gap Sibling Communication through Technology Probes

Stuti Arora, Qiao Jin, Svetlana Yarosh

Abstract

Large age differences and geographic distance between siblings can often hinder the development of close relationships, as they may have different social circles and schedules. Younger siblings may also have limited access to digital technology, which can further complicate communication and interaction between them. In this paper, we developed two technology probes, Haptic Bubble and Emoji Board. Both of these systems utilize embodied interaction, which has been shown to be an effective way to engage children in remote communication. Our work focused on three main goals through the development and preliminary study: the system design goal involved testing the feasibility of the embodied design approach, the empirical goal was to collect information on how siblings use embodied communication technology and the design goal was to inspire new kinds of technology to support large gap siblings' needs.

Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (UMAP Adjunct 2023), GMAP Workshop

Less Can Be More: Exploring Population Rating Dispositions with Partitioned Models in Recommender Systems

Ruixuan Sun, Ruoyan Kong, Qiao Jin, Joseph A. Konstan

Abstract

In this study, we partition users by rating disposition - looking first at their percentage of negative ratings, and then at the general use of the rating scale. We hypothesize that users with different rating dispositions may use the recommender system differently and therefore the agreement with their past ratings may be less predictive of the future agreement. We use data from a large movie rating website to explore whether users should be grouped by disposition, focusing on identifying their various rating distributions that may hurt recommender effectiveness. We find that such partitioning not only improves computational efficiency but also improves top-k performance and predictive accuracy.

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 2023)

Socio-technical Opportunities in Long-Distance Communication Between the Siblings with a Large Age Difference

Qiao Jin, Ye Yuan, Svetlana Yarosh

Best Paper Honorable Mention

Abstract

Siblings play a crucial and long-lasting role in family connections and relationships. However, with the older sibling transitioning out of their parental home, maintaining a close sibling relationship can be challenging, especially if siblings have a large age difference. We conducted a diary and interview study with nine families in China which have spaced siblings, to identify design opportunities for technology to better support their communication and connection needs. We contribute to the HCI community in three aspects. First, we contribute an empirical understanding of current communication patterns from distributed families with large age gap siblings in China. Second, we identify current facilitation roles, practices, and challenges regarding sibling relationships from different stakeholders' perspectives. Last but not least, we present technological opportunities for supporting the large-gap sibling relationship, informing directions for future research and design for distributed families.

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 2023)

Collaborative Online Learning with VR Video: Roles of Collaborative Tools and Shared Video Control

Qiao Jin, Yu Liu, Ruixuan Sun, Chen Chen, Puqi Zhou, Bo Han, Feng Qian, Svetlana Yarosh

Abstract

Virtual Reality (VR) has a noteworthy educational potential by providing immersive and collaborative environments. As an alternative but cost-effective way of delivering realistic environments in VR, using 360-degree videos in immersive VR (VR videos) received more attention. Although many studies reported positive learning experiences with VR videos, little is known about how collaborative learning performs on VR video viewing systems. In this study, we implemented two collaborative VR video viewing modes based on the way of group video control, synchronized or shared (Sync mode) and non-synchronized or individual (Non-sync mode) video control, against a conventional VR video viewing setting (Basic mode). We conducted a within-subject study (N = 54) in a lab-simulated remote learning environment. Our results show that collaborative VR video modes (Sync and Non-sync mode) significantly improve learning outcomes and satisfaction compared to the Basic mode.

IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2023)

Volumivive: An Authoring System for Adding Interactivity to Volumetric Video

Qiao Jin, Yu Liu, Puqi Zhou, Bo Han, Svetlana Yarosh, Feng Qian

Abstract

Volumetric video is a medium that captures the three-dimensional (3D) shape and movement of real-life objects or people. However, pre-recorded volumetric video is limited in terms of interactivity. We introduce a novel authoring system called Volumivive, which enables the creation of interactive experiences using volumetric video, enhancing the dynamic capabilities and interactivity of the medium. We provide four interaction methods that allow users to manipulate and engage with digital objects within the volumetric video. These interactive experiences can be used in both augmented reality (AR) and virtual reality (VR) settings, providing users with a more immersive and interactive experience.

2022

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI 2022)

How Will VR Enter University Classrooms? Multi-stakeholders Investigation of VR in Higher Education

Qiao Jin, Yu Liu, Svetlana Yarosh, Bo Han, Feng Qian

Abstract

VR has received increased attention as an educational tool and many argue it is destined to influence educational practices, especially with the emergence of the Metaverse. Most prior research on educational VR reports on applications or systems designed for specified educational or training objectives. However, it is also crucial to understand current practices and attitudes across disciplines, having a holistic view to extend the body of knowledge in terms of VR adoption in an authentic setting. Taking a higher-level perception of people in different roles, we conducted a qualitative analysis based on 23 interviews with major stakeholders and a series of participatory design workshops with instructors and students. We identified the stakeholders who need to be considered for using VR in higher education, and highlighted the challenges and opportunities critical for VR current and potential practices in the university classroom. Finally, we discussed the design implications based on our findings. This study contributes a detailed description of current perceptions and considerations from a multi-stakeholder perspective, providing new empirical insights for designing novel VR and HCI technologies in higher education.

2021

ACM Symposium on User Interface Software and Technology (UIST 2021), Student Innovation Contest

MoMoBot: Social Robots for Remote Communication

Qiao Jin, Irene Ye Yuan

Best Toio Award