Mnemonic Tracing: Using Eye Gaze to Search for Visual Memories

How it Works

Mnemonic Tracing turns your eye movements into a way of searching your memory.

First, when you see an image, your gaze naturally traces important details—objects, shapes, and regions that stand out to you. We capture this as a soft “attention map,” representing how your eyes explored the scene.

Later, when you try to recall the image, you trace it again from memory on a blank screen. Even without the image present, your eyes tend to follow a similar pattern.

Our GAMR (Gaussian Attention Map Retrieval) algorithm then compares these two traces—how you looked at the image and how you remembered it. It measures how closely they match and ranks the most likely images based on this similarity.

No training, calibration, or personalization is required. The system works by leveraging a natural property of human memory—how perception and recall are inherently linked through eye movement.

What We Learned

We evaluated Mnemonic Tracing through a controlled pilot study with 11 participants, exploring how people can retrieve visual memories using gaze alone. The results highlight both the technical feasibility and the human experience of this new interaction:

Performance: The training-free GAMR algorithm achieved 51.2% Top-3 accuracy, significantly above chance (10%), demonstrating that gaze patterns carry meaningful signals for retrieval.

Perceived Usefulness:  M = 3.58/5 (SD = 0.82) (TAM survey)

No Training Required: The system works without any calibration or user-specific training, relying purely on the natural alignment between how we see and how we remember.

Embodied Recall: Participants naturally engaged in tracing behaviors, often reconstructing scenes object-by-object—turning recall into an active, embodied process rather than passive remembering.

Cognitive Experience: Many described the interaction as reflective and attention-driven, with some noting it encouraged deeper awareness of visual details in their surroundings.

Challenges: Users found it difficult to reproduce exact spatial layouts, often simplifying or scaling down scenes. Recall was also sequential and slower than expected, revealing a gap between mental imagery and precise spatial reproduction.

Perhaps most tellingly, participants described the experience as “a tension between involuntary movement and intentional control,” “surprisingly mindful,” and at times “effortful but engaging”—suggesting a new form of interaction that sits between perception, memory, and action.

What This Enables

Mnemonic Tracing points toward a new class of memory interfaces—ones that don’t rely on words, but on how we naturally see and recall.

With the rise of everyday smartglasses, this interaction can move beyond the lab into continuous, real-world use—where memory retrieval becomes seamless, embodied, and always available.

This opens pathways for:

Memory Aids — Supporting recall and visual learning by allowing people to retrieve what they’ve seen without needing to describe it.

Image & Video Retrieval — Searching personal photo archives or navigating video timelines by tracing remembered scenes.

Spatial Navigation — Reliving past experiences by following the movement of objects and visual cues in memory.

Accessibility & Forensics — Enabling recall for users who cannot speak or use their hands, or in situations where verbal descriptions fail (e.g., remembering a face).

More Details

Mnemonic Tracing will be presented at ACM CHI 2026 on April 16th, with our full research available here. We're also releasing the system implementation as open source at https://github.com/mlfluidinterfaces/mnemonic-tracing.