Eye Tracking And Pupil Movement Authentication In Python — Complete Documentation & Simulation

Overview & System Context

Eye Tracking And Pupil Movement Authentication In Python: Comprehensive Web-Sourced Technical Investigation Exhaustive Research Dossier · Pure Web-Scraped Content: ~2561 Words · Real Software Screenshots: 1 1. Executive Overview & Technical Background pyEyeTrack is a python-based pupil-tracking library. The library tracks eyes with the commodity webcam and gives a real-time stream of eye coordinates. It provides the functionality of eye-tracking and blink detection and encapsulates these in a generic interface that allows clients to use these functionalities in a variety of use-cases.

You may want to use a conda virtual environment to avoid mix up with system dependencies. For any issues regarding PyEyeTrack, contact the PyEyeTrack support at pyeyetrack@gmail.com Kanchan Sarolkar, Kimaya Badhe, Neha Chaudhari, Samruddhi Kanhed and Shrirang Karandikar. Figure 1: Figure 1: Verified System Architecture & Pipeline Workflow PyEyeTrack is a python-based pupil-tracking library. The library tracks eyes with the commodity webcam and gives a real-time stream of eye coordinates.

This is a Python library (3.10+) that provides a webcam-based eye tracking system . It gives you the exact position of the pupils and the gaze direction, in real time. Trouble installing dlib? The wheel usually installs fine. If pip tries to compile it from source, install CMake first: brew install cmake (macOS) or sudo apt install cmake build-essential (Ubuntu).

Pass the frame to analyze (numpy.ndarray). If you want to work with a video stream, you need to put this instruction in a loop, like the example above. Returns a number between 0.0 and 1.0 that indicates the horizontal direction of the gaze. The extreme right is 0.0, the center is 0.5 and the extreme left is 1.0.

Returns a number between 0.0 and 1.0 that indicates the vertical direction of the gaze. The extreme top is 0.0, the center is 0.5 and the extreme bottom is 1.0. Your suggestions, bugs reports and pull requests are welcome and appreciated. You can also starring II the project! If the detection of your pupils is not completely optimal, you can send me a video sample of you looking in different directions.

I would use it to improve the algorithm. This project is released by Antoine Lamé under the terms of the MIT Open Source License. View LICENSE for more information. YouâIIll also need dlibâIIs pre-trained facial landmark model ( shape_predictor_68_face_landmarks.dat ), which detects 68 key points on the face (including eyes). Download it from dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 , extract it, and save it in your project folder.

Challenges : Lighting (avoid backlighting), glasses, and varying eye colors can affect accuracy. WeâIIll use thresholding and contour detection to isolate pupils, as they are consistently the darkest part of the eye. 2. System Architecture & Core Working Principles Explanation : cv2.VideoCapture(0) initializes the webcam. Frames are read in BGR format (OpenCV default), so we convert to grayscale to reduce computational load.

To focus on the eyes, we first detect the face. WeâIIll use dlibâIIs HOG-based face detector, which is robust to varying face orientations. Explanation : detector(gray) returns a list of dlib.rectangle objects, each representing a detected face. We draw a green rectangle around the face for visualization.

Using dlibâIIs 68-point facial landmarks, we can precisely locate the eyes. The eyes correspond to landmarks 36âII41 (left eye) and 42âII47 (right eye). Explanation : Landmarks are used to crop the eye regions ( left_eye_roi and right_eye_roi ), which weâIIll process next to detect pupils.

Pupils are dark and circular, but noise (e.g., eyelashes, reflections) can interfere. Preprocessing enhances contrast and reduces noise. Pupils are the largest dark contours in the eye region. WeâIIll find contours in the thresholded image and select the largest one as the pupil.

Explanation : cv2.findContours finds connected white regions (pupil candidates). The largest contour is selected, and its centroid (center) is computed using image moments. Pupil positions vary between users. Calibration sets thresholds for âIIleft,âII âIIright,âII âIIup,âII and âIIdownâII commands by having the user look at reference points.

Explanation : During calibration, the user looks at on-screen markers (e.g., âIILook leftâII), and their pupil centroids are stored. These values define the boundaries for control commands. Using calibrated thresholds, map pupil centroid positions to robot commands (e.g., âIIleft,âII âIIrightâII). YouâIIve built a system to track eye pupils and control a robotâIIno expensive hardware required! This project demonstrates the power of computer vision and opens doors to advanced applications like assistive tech for motor-impaired users or hands-free gaming.

Next Steps : Add blink detection for clicks, use 3D eye tracking with stereo cameras, or integrate machine learning for better pupil detection in challenging conditions. PyEyeTrack is a python-based pupil-tracking library. The library tracks eye movements with commodity hardware, such as a laptop webcam, and gives a real-time stream of eye coordinates. It provides the functionality of eye-tracking and blink detection and encapsulates these in a generic interface that allows clients to use these functionalities in a variety of use-cases.

The goal has been to make the library generic. A user can provide any interface: a static image, text, video, and call upon PyEyeTrack to capture eye-movement data. The default is to measure eye-movement data captured by a live camera feed, a saved video file can also be used as input. PyEyeTrack can optionally save audio and video as an aid to debugging and development.

3. Software Implementation, Frameworks & Protocols Use PyEyeTrack to develop applications that can be controlled by eye movements or blinks. Or use it to track eye movements as an aid to medical diagnostics. PyEyeTrack has three modules - EyeTracking, AudioVideo recording, and the main interface (along with the data handling class). The EyeTracking module is the core of the library that performs eye tracking and blink detection. This module is responsible for accessing the webcam to track eyes and write it to a CSV. AudioVideo Recording module records audio and video. The third module is the primary interface to PyEyeTrack - The PyEyeTrackRunnerClass.

This class provides the user with an interface to execute the functionalities of the library. The user can specify the UI, the eye-tracking functionality, the destination folder and exploit the functions of this library by using various combinations with the following call- pyEyeTrack_runner(UI = False, UI_file_name = “User_ImageUI_EscExit”, pupilTracking = False, blinkDetection = False, video_source = 0, eyeTrackingLog = True, eyeTrackingFileName = ‘EyeTrackLog’, videoRecorder = False, videoName = ‘video’, audioRecorder = False, audioName = ‘audio’, destinationPath = ‘/Output’) UI_file_name(str, optional): This parameter takes the file name of the UI. Default: “User_ImageUI_EscExit”.

pupilTracking(bool, optional): This parameter enables the user to run pupil tracking. Default: False. blinkDetection(bool, optional): This parameter enables the user to run blink detection. Default: False. video_source(int/str, optional): This parameter takes either device index or a video file as input. Default: 0. eyeTrackingLog(bool, optional): This parameter enables the user to generate a CSV of pupil tracking/ blink detection. Default: True.

eyeTrackingFileName(str, optional): This parameter takes the file name for the CSV. Default: ‘EyeTrackLog’. videoRecorder(bool, optional): This parameter enables the user to record video. Default: False. videoName(str, optional): This parameter enables the user to specify the filename with which the recorded video is to be saved. Default: ‘video’.

audioRecorder(bool, optional): This parameter enables the user to record audio. Default: False. audioName(str, optional): This parameter enables the user to specify the filename with which the recorded audio is to be saved. Default: ‘audio’. destinationPath(str, optional): The parameter enables the user to specify the location of the output files. Default: ‘/Output’.

4. Practical Deployment, Analysis & Case Studies The Data Handling class is responsible for handling the real-time pupil locations and blinks dynamically. This class implements a queue that the user has access to get the eye location or the blink details dynamically. This functionality gives the user the freedom to use eye-tracking data in real-time as per their wish. The user can use the following functions to handle the data: add_data(data) : This function allows the user to add data to the dynamic queue. get_data() : This function returns the data elements of the queue. is_empty() : This function checks if the queue is empty. search_element(key) : This function is used to search if a specified key To make this clear, a basic example that makes use of the library to track the eyes of the user reading a text UI is described below.

pyEyeTrack_runner(UI = True, UI_file_name = “Ex_1_SampleTextUI”, pupilTracking = True, eyeTrackingLog = True, eyeTrackingFileName = ‘User_1’) This example uses the pupil tracking functionality of the library. This example tracks the eyes of the user as the user reads the text displayed on the screen. The text UI is specified by the user. The user can set the destination path. In this case, as the destination path is not specified, the pupil coordinates CSV will be stored in the default ‘Output’ folder.

Library requirements: The library needs FFmpeg executable file in the working directory for the audio-video syncing to work. How to merge the audio and video? 1. Download ffmpeg.exe from here 2. Paste the .exe file to the same folder as the audio,video files. 3. Open command prompt in that folder. 4. Command - 'ffmpeg -y -i audio.wav -r 30 -i video.avi -filter:a aresample=async=1 -c:a flac -c:v copy avoutput.mkv' Reference Dependencies: dlib 19.4 conda install -c conda-forge dlib=19.4 You may want to use a conda virtual environment to avoid mix up with system dependencies.

UI requirements: The library can run any UI developed in python. The user needs to ensure that the UI has the main function. The library takes the name of the UI as input and runs the main function in it. For any issues regarding PyEyeTrack, contact the PyEyeTrack support at pyeyetrack@gmail.com Kanchan Sarolkar , Kimaya Badhe , Neha Chaudhari , Samruddhi Kanhed and Shrirang Karandikar In this tutorial we will explore simultaneously recorded eye-tracking and EEG data from a pupillary light reflex task. We will combine the eye-tracking and EEG data, and plot the ERP and pupil response to the light flashes (i.e. the pupillary light reflex).

As usual we start by importing the modules we need and loading some example data : eye-tracking data recorded from SR research’s '.asc' file format, and EEG data recorded from EGI’s '.mff' file format. We’ll pass create_annotations=["blinks"] to read_raw_eyelink() so that only blinks annotations are created (by default, annotations are created for blinks, saccades, fixations, and experiment messages).

The info structure of the eye-tracking data tells us we loaded a monocular recording with 2 eyegaze channels (x- and y-coordinate positions), 1 pupil channel, 1 stim channel, and 3 channels for the head distance and position (since this data was collected using EyeLink’s Remote mode). By default, EyeLink files will output ocular events (blinks, saccades, and fixations), and experiment messages. MNE will store these events as mne.Annotations . Ocular annotations contain channel information in the 'ch_names' key. This means that we can see which eye an ocular event occurred in, which can be useful for binocular recordings: EyeLink .asc files can also include calibration information. MNE-Python can load and visualize those eye-tracking calibrations, which is a useful first step in assessing the quality of the eye-tracking data. read_eyelink_calibration() will return a list of Calibration instances, one for each calibration. We can index that list to access a specific calibration.

5. Performance Evaluation & Extended Technical Observations Calibrations have dict-like attribute access; in addition to the attributes shown in the output above, additional attributes are 'positions' (the x and y coordinates of each calibration point), 'gaze' (the x and y coordinates of the actual gaze position to each calibration point), and 'offsets' (the offset in visual degrees between the calibration position and the actual gaze position for each calibration point). Below is an example of how to access these data: Let’s plot the calibration to get a better look. Below we see the location that each calibration point was displayed (gray dots), the positions of the actual gaze (red), and the offsets (in visual degrees) between the calibration position and the actual gaze position of each calibration point.

If you supply the eyetracker monitor’s screen resolution to Calibration , plot() will automatically set the canvas bounds accordingly. EyeLink stores eyegaze positions in pixels, and pupil size in arbitrary units. MNE-Python expects eyegaze positions to be in radians of visual angle, and pupil size to be in meters. We can convert the eyegaze positions to radians using convert_units() .

In addition to the screen resolution, we need to supply the eyetracker screen size and screen distance to our Calibration object, before calling the convert units function. Let’s plot the raw eye-tracking data. Since we did not convert the pupil size to meters, we’ll pass a custom dict into the scalings argument to make the pupil size traces legible when plotting.

Naturally, there are blinks in our data, which occur within "BAD_blink" annotations. During blink periods, eyegaze coordinates are not reported, and pupil size data are 0 . SR Research EyeLink files already ship blink annotations, which is why these were created when we loaded the data with create_annotations=["blinks"] . Many other eye-trackers do not. For those, you can detect blinks from the pupil channel with find_blinks() , which annotates segments where the pupil signal drops out. EyeLink encodes dropouts as 0 (hence dropout_value=0 below; the default looks for np.nan ). The function returns the annotations rather than modifying raw in place, so you can inspect or edit them before attaching them with set_annotations() .

We don’t want these blink artifacts biasing our analysis, so we have two options: drop the blink periods during epoching, or interpolate the missing data. For this tutorial we’ll interpolate, passing (0.05, 0.2) to interpolate_blinks() to expand the interpolation window 50 ms before and 200 ms after each blink, so that the noisy data surrounding the blink is also interpolated.

By default, interpolate_blinks() , will only interpolate blinks in pupil channels. Passing interpolate_gaze=True will also interpolate the blink periods of the eyegaze channels. Be aware, however, that eye movements can occur during blinks which makes the gaze data less suitable for interpolation. In this experiment, a photodiode attached to the display screen was connected to both the EEG and eye-tracking systems.

The photodiode was triggered by the the light flash stimuli, causing a signal to be sent to both systems simultaneously, signifying the onset of the flash. The photodiode signal was recorded as a digital input channel in the EEG and eye-tracking data. MNE loads these data as a stim channel . We’ll extract the flash event onsets from both the EEG and eye-tracking data, as they are necessary for aligning the data from the two recordings.

The output above shows us that both the EEG and EyeLink data used event ID 2 for the flash events, so we’ll create a dictionary to use later when plotting to label those events. In this dataset, eye-tracking and EEG data were recorded simultaneously, but on different systems, so we’ll need to align the data before we can analyze them together. We can do this using the realign_raw() function, which will align the data based on the timing of the shared events that are present in both Raw objects. We’ll use the shared photodiode events we extracted above, but first we need to convert the event onsets from samples to seconds. Once the data have been aligned, we’ll add the EEG channels to the eye-tracking raw object.

Now let’s extract epochs around our flash events. We should see a clear pupil constriction response to the flashes. For this experiment, the participant was instructed to fixate on a crosshair in the center of the screen. Let’s plot the gaze position data to confirm that the participant primarily kept their gaze fixated at the center of the screen. Harvested Online Documentation & Reference Sources • https://github.com/algoasylum/pyEyeTrack • https://github.com/antoinelame/GazeTracking • https://www.w3reference.com/blog/track-eye-pupil-position-with-webcam-opencv-and-python/ • https://pyeyetrack.algoasylum.com/ • https://mne.tools/stable/auto_tutorials/preprocessing/90_eyetracking_data.html