This project models eye-gaze trajectories as continuous-time dynamics with a neural ordinary differential equation that treats saccades as discrete jump events. Training uses COCO-Search18, a laboratory-quality dataset of goal-directed visual search (10 participants, 18 target categories, 6,202 images, approximately 300,000 fixations) and, where helpful, its listing within the Massachusetts Institute of Technology and University of Tübingen Saliency Benchmark for benchmark-ready evaluation. The learned vector field is conditioned on image features from a convolutional neural network and on the target label to predict the timing and location of the next fixation; evaluation covers average displacement error and final displacement error, distributional fidelity of saccade amplitudes and durations, and task outcomes such as time to target.
This project implements a piecewise neural ordinary differential equation: continuous dynamics during fixations and smooth pursuit with discrete jumps at saccade onsets, and compares against legacy models, such as autoregressive models, hidden Markov models, and long short-term memory networks and transformer networks. An optional extension replaces the jump mechanism with a neural jump stochastic differential equation to model stochastic event timing.
References:
COCO-Search18 dataset:
https://sites.google.com/view/cocosearch/
Neural ODE:
https://arxiv.org/abs/2103.12413
Virtual-Eye Project: https://www.simula.no/research/projects/virtual-eye-learning-human-eye-scanpaths-optimal-autonomous-search