Nicu Monitoring System

Ai Device

Jyothi Hariharan, Rahul Rama Varior, and Sunil Karunakaran Ignitarium Technology Solutions Private Limited, Bangalore, India Abstract. As road accident cases are increasing due to the inattention of the driver, automated driver monitoring systems (DMS) have gained an increase in acceptance. In this report, we present a real-time DMS sys- tem that runs on a hardware-accelerator-based edge device. The system consists of an InfraRed camera to record the driver footage and an edge device to process the data. To successfully port the deep learning models to run on the edge device taking full advantage of the hardware acceler- ators, model surgery was performed. The final DMS system achieves 63 frames per second (FPS) on the TI-TDA4VM edge device.

Figure: System Model & Simulation Flow for Nicu Monitoring System

Keywords: DMS, Driver Monitoring System, TDA4VM, TIDL, Auto-

Introduction

Road accidents are increasing due to the inattention of the driver. Around the world, there is an increase in the acceptance of automated systems to alert the driver in the event of a distraction from the driving task. For instance, Euro New Car Assessment Programme (NCAP) standards and the European Commission (EC) regulation mandates Driver Monitoring Systems (DMS) technology from 2023. The U.S. National Transportation Safety Board (NTSB) recommends us- ing DMS in semi-autonomous cars. DMS built using Artificial intelligence (AI) technology is proving to be an effective safety feature and has gained much at- tention in the automotive industry in recent years. In this article, we present the in-house research and development efforts to build a camera-based real-time DMS system that runs on an edge device.

Figure: System Model & Simulation Flow for Nicu Monitoring System

Safety features in automobiles should allow sufficient response time for the driver to take necessary actions. Hence, the data must be processed, and the driver must be alerted in real time if the driver is distracted or drowsy. While numerous topologies (including CPU-only, CPU + GPU, CPU + Accelerator etc.) exist, the specific compute-architecture selected will usually have a direct bearing on the cost and power consumption of the chosen edge device. Deploying Deep Learning (DL) based AI systems on low-cost edge devices can be challeng- ing due to memory limitations and low processing capacity. Moreover, porting the Deep Learning models to run on the edge device to take full advantage of the specialized AI hardware accelerators can be challenging due to a host of reasons.

2

J. Hariharan et al. All required DL operators and parameter ranges maybe not be supported by the vendor’s Software Development Kit (SDK). In many cases, support may be available in the SDK but the implementation may not be highly optimized for the underlying hardware.

In this article, we present our R&D efforts to build a camera based real- time DMS system that runs on a hardware-accelerator-based edge device. First, the high-level architecture of the DMS is presented along with the details of the individual components of the system. The challenges and solutions to successfully port the model to run on an exemplary edge device hardware accelerator (viz.

Texas Instruments TDA4VM) are explained in the next section. Finally, we present the performance improvements in terms of inference time and frames per second (FPS) that we achieved after successfully porting the model to run on the hardware accelerator.

High-Level Architecture Of Dms

The DMS consists of an infrared camera from Leopard Imaging, that is used to capture video data. The collected data is processed by the AI stack which runs on the edge AI device. The results, for the purpose of demonstration, are displayed on a standard display device. Note that the display device is only for visualization purposes and is not part of the end system. Fig 1 shows the components of the DMS. Subsequent sections describe each of the components in detail.

Fig. 1. Components of DMS (a) The LI-USB30-AR0144IVEC-GMSL2-055H IR camera

Infrared Camera

To capture driver footage, an infrared (IR) camera LI- USB30-AR0144IVEC-GMSL2-055H from Leopard Imaging (LI) is mounted on the dashboard inside the driver cabin. The advantages of the LI IR camera are twofold. First, the camera module has IR illumination which enables low-light and night vision. The IR filter in the camera module also prevents interference due to ambient lighting. Second, IR cameras enable ‘see-through’ of certain sun- glasses so that the eye pupil location can be accurately captured. For more details on the camera specifications and its datasheet, interested readers can refer to the product datasheet.

Ai Stack

The AI stack consists of the edge device and the perception system or the software application. Edge device : The edge device used is TDA4VM from Texas Instruments.

TDA4VM features a dual 64-bit Arm® Cortex®-A72 microprocessor subsys- tem, a C7x floating point vector DSP of up to 80 GFLOPS, and a deep-learning Matrix Multiply Accelerator (MMA) of up to 8 TOPS (8-bits). The specialized hardware accelerator MMA processes much of the deep learning workloads and is optimized for 8-bit precision (int8) and 16-bit (int16) precision integer compu- tations. Interested readers can find more information about the product in the TDA4VM datasheet. We used the SDK version 8.02.00.

Perception system : The perception system consists of a deep learning- based face detection model followed by a face landmark model. Individual frames from the video input are passed through the face detection algorithm to detect the driver’s face. The detected face is cropped and passed as input for the face landmark model. The face landmark algorithm estimates key facial points such as the face silhouette, the iris points, eye points, eyebrow points, upper and lower lips points, nose bridge, and nose bottom. These facial landmark points are used to estimate the head pose, eye closure duration, and eye gaze of each eye, and to detect whether the driver is yawning or not. The yawn results, eye closure duration, the eye gaze, and the head pose of the driver are used to infer the state of the driver, i.e., whether the driver is distracted, drowsy, or alert. The pipeline stages of the perception system are shown in Fig 2.

Face detection deep learning models: For face detection, we conducted experiments with Mediapipe face detection model BlazeFace and YOLOX- tiny model available in the TI edge AI vision model-zoo.

Face landmark deep learning models: For face landmark estimation, we conducted experiments with the Google Mediapipe face mesh model and the Practical Facial Landmark Detector (PFLD) model. In the following section, the model porting and optimization efforts are explained.

Model Porting For Tda4Vm

The TDA4VM is equipped with the MMA which is a hardware accelerator that processes deep learning workloads. However, it is possible that not all the deep

4

J. Hariharan et al.

Fig. 2. Pipeline Stages Of The Perception System

learning operators or operator configurations are supported for execution in the MMA. The unsupported operators, in such cases, will normally be executed in the ARM CPU core resulting in a reduction in the overall inference speed of the deep learning model. To overcome this, the pre-trained models will have to be adapted so that their functionality is not affected and is fully executed on the accelerator. The following section describes our efforts to port the model so that it is fully offloaded to the MMA.

While loading the DL models to run on the MMA, the input data to the model will be copied from the CPU to the accelerator memory. In case an opera- tor in the DL model is not supported by the device hardware accelerator library, the operation will be performed by the ARM CPU. The input data to the corre- sponding layer will have to be copied from the accelerator memory to the CPU memory for executing the layer. After the layer is executed, the outputs need to be copied back to the hardware accelerator memory for processing the rest of the layers. The data transfer between the CPU and the accelerator is expensive and will result in suboptimal inference speed.

To take full advantage of the hardware accelerator, all layers of the model must be offloaded to the MMA. Therefore, unsupported layers/operators in the model’s frozen graph must be replaced/removed (model graph surgery or model surgery) with functionally accurate layers/operators that are supported in the MMA. In order to perform the model surgery, we develop a model graph trans- lation library (or toolkit). The model graph translation library helps to create a trainable model from a frozen tflite or ONNX graph and the surgery can be performed on the trainable model. The benefits of using this library are:

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.