Flash Glucose Monitor

Arxiv:2204.03376V2 [Cs.Lg] 5 May 2023

Offline Reinforcement Learning for Safer Blood Glucose

Control In People With Type 1 Diabetes

Harry Emersona,∗, Matthew Guyb,, Ryan McConvillea,

Abstract

The widespread adoption of effective hybrid closed loop systems would rep- resent an important milestone of care for people living with type 1 diabetes (T1D). These devices typically utilise simple control algorithms to select the optimal insulin dose for maintaining blood glucose levels within a healthy range. Online reinforcement learning (RL) has been utilised as a method for further enhancing glucose control in these devices. Previous approaches have been shown to reduce patient risk and improve time spent in the tar- get range when compared to classical control algorithms, but are prone to instability in the learning process, often resulting in the selection of unsafe actions. This work presents an evaluation of offline RL for developing ef- fective dosing policies without the need for potentially dangerous patient interaction during training. This paper examines the utility of BCQ, CQL and TD3-BC in managing the blood glucose of the 30 virtual patients avail- able within the FDA-approved UVA/Padova glucose dynamics simulator.

When trained on less than a tenth of the total training samples required by online RL to achieve stable performance, this work shows that offline RL can significantly increase time in the healthy blood glucose range from 61.6±0.3% to 65.3±0.5% when compared to the strongest state-of-art base- line (p < 0.001). This is achieved without any associated increase in low blood glucose events.

May 8, 2023

common and challenging control scenarios such as incorrect bolus dosing, irregular meal timings and compression errors. The code for this work is available at: https://github.com/hemerson1/offline-glucose.

Keywords:

reinforcement learning, type 1 diabetes, glucose control,

1. Introduction

Type 1 diabetes (T1D) is an autoimmune disease characterised by an insufficiency of the hormone insulin, which is required for blood glucose reg- ulation. People with T1D must regularly monitor their blood glucose levels and estimate the correct dosage of insulin and carbohydrate intake to avoid dangerous instances of low and high blood glucose.

This Includes Taking

bolus insulin to account for ingested meal carbohydrates, in addition to ad- justing basal insulin to account for fluctuations between meals. Hybrid closed loop systems provide an opportunity for people with T1D to automatically regulate their basal insulin dosing [1, 2, 3]. These devices consist of an in- sulin pump connected to a continuous glucose monitor (CGM) by a control algorithm. The CGM measures the blood glucose level of the user and the control algorithm uses the CGM data to instruct the insulin pump to deliver the required dosage. This process repeats at regular intervals and corrects for deviations in blood glucose from some target blood glucose range or value.

Trials of these devices in adults and pediatrics have shown a significant as- sociation between the use of hybrid closed loop systems and improvements to time spent in the recommended blood glucose range [4, 5, 6]. Hartnell et al. provides a detailed overview of existing closed loop devices and their functionality .

The majority of commercially available hybrid closed loop systems utilise predictive integral derivative (PID) controllers or model predictive controllers (MPC) . These algorithms are robust and easily interpretable, but limit the efficacy of the devices. PID algorithms are prone to overestimating in- sulin doses following meals and are unable to easily incorporate additional factors which affect blood glucose, such as insulin activity, time of day and exercise [9, 10]. In contrast, MPCs typically utilise linear or simplified mod- els of glucose dynamics, which are unable to capture the full complexity of the task [11, 12]. Reinforcement learning (RL) has been proposed as a means of addressing this problem; through which a decision-making agent

2

learns the optimal sequence of actions to take in order to maximise some concept of reward. In glucose control, RL algorithms have demonstrated an ability to learn sophisticated and personalised control policies for individual patients. These policies often outperform their PID and MPC counterparts when trained and evaluated in simulators of glucose dynamics, but are im- practical for clinical use in their present state [13, 14, 15]. Current approaches predominantly utilise online RL algorithms, which require interaction with a patient or simulator during training to develop control policies and learn via a process akin to trial and error. These agents typically start with a poor un- derstanding of their environment and are prone to learning instability [16, 17], both of which could feasibly contribute to the selection of dangerous insulin doses. This facet of online RL limits its utility in real-world hybrid closed loop systems; highlighting the necessity for methods capable of learning ac- curate dosing policies from clinically obtainable quantities of glucose data without the associated risk.

This work presents a proof-of-concept in silico study on the use of offline RL for glucose control, in which an RL agent learns without environmen- tal interaction during training and instead learns from a static dataset of demonstrations collected under another agent. This entails a rigorous anal- ysis of the offline RL algorithms: batch constrained deep Q-learning (BCQ) , conservative Q-learning (CQL) and twin delayed deep deterministic policy gradient with behavioural cloning (TD3-BC) in their ability to develop safe and high performing insulin dosing strategies in hybrid closed loop systems. The presented algorithms are trained and tested across a co- hort of 30 virtual patients (10 children, 10 adolescents and 10 adults) and their performance and sample-efficiency is scrutinised with respect to the cur- rent strongest online RL and control baselines. Practical limitations such as missing CGM data and suboptimally set PID parameters are also explored, as these are common features of real-world blood glucose data. To ensure the reliable and effective operation of offline RL in the worst-case scenarios, safety is scrutinised across a range of realistic and common control events.

This includes overdosing in mealtime insulin, sporadic and irregular meal events and erroneous compression lows caused by force on the CGM inser- tion site. This work shows that offline RL can yield more effective and safer insulin dosing policies without the patient interaction required by prior RL approaches. Furthermore, this method also utilises significantly smaller sam- ples of data making it more applicable for use in real patients. The presented results provide a foundational overview of the advantages and limitations of

3

using offline RL in glucose control tasks and will provide a reference for fu- ture research seeking to integrate offline RL in hybrid closed loop systems before trialling on real-world patients in a controlled setting.

2. Related Work

Offline RL is an area of increasing interest in healthcare due to the safety concerns associated with incorrect decision-making . Previous healthcare research has focused on developing lung cancer protocols from samples of historical data , identifying optimal recommendations for sepsis treatment and facilitating RL policy evaluation in an offline medical setting .

Despite the prevalence of offline RL in other domains of medicine, its use in glucose control has been limited . Javad et al. used a q-learning algo- rithm trained offline on samples of clinical data to select the optimal daily basal dose for patients of a given demographic . Shi et al. presented an offline q-learning approach trained on the OhioT1DM dataset for selecting discrete doses of basal insulin at hourly intervals . Similarly, Li et al. used a model-based RL algorithm to learn the blood glucose dynamics of patients recovering from diabetic ketoacidosis . This learned model was then used to train an online q-learning algorithm to select the optimal basal dose at three hour intervals over a 24-hour period. Although these approaches pre- sented methods for learning dosing policies offline, the timeframe over which they act would be insufficient for managing the short-term blood glucose fluc- tuations associated with meals and exercise. The most significant application of offline RL for hybrid closed loop systems was presented in Fox, in which the methods BCQ and bootstrapping error accumulation reduction (BEAR) were compared to online RL approaches trained on an imperfect simulator of a single adult patient . Analysis showed that although offline RL was capable of developing competent control policies, the achieved performance was less than that obtained using an online approach trained on an imperfect simulator.

Blood glucose forecasting is an integral aspect of T1D management , consequently significant research has been focused on extending established control algorithms using blood glucose prediction models dervied via super- vised learning. This has included using quantile regression to predict upper and lower bounds on future blood glucose values , bolstering fuzzy logic controllers with neural network prediction and leveraging a pair neural networks to forecast blood glucose and select insulin doses based on the pre-

4

dictions . A number of these approaches have also been experimentally validated within animal trials [34, 35]. Of particular note, Chen et al. trains a behavioural cloning agent on the insulin doses of an MPC demonstrator; providing a sample-efficient alternative of learning insulin dosing strategies without the risk of dangerous action selection during training. Supervised learning has also been used in hybrid closed loop systems for functions sec- ondary to insulin dosing, such as hypoglycemia prediction and detecting unreported carbohydrate consumption .

Within online RL, several attempts have been made to address concerns around safety and learning instability in glucose control. Fox et al. employed a transfer learning approach to develop dosing policies from a general patient population before fine-tuning them on target patients . Lim et al. used a PID controller to guide an online RL algorithm in the early stages of its learning; progressively introducing a greater proportion of RL agent actions . Zhu et al. developed an online RL algorithm capable of integration with a dual hormone pump, allowing control of both insulin and glucagon dosing and hence for low blood glucose corrections to be made through glucagon in- fusion . These approaches all showed comparable or reduced time spent in the potentially dangerous low blood glucose range when contrasted with MPC and PID algorithms, but are limited by their reliance on glucose dy- namics simulators. These simulators represent a simplification of true blood glucose dynamics; with almost all unable to incorporate common events such as exercise, stress and illness. In moving towards use in real-world hybrid closed loop systems, online RL performance is likely to deteriorate as la- tent variables start to influence the environment . In addition, glucose dynamics simulators allow for unlimited training data generation. Of the ap- proaches highlighted which explicitly stated their sample size, the algorithms used 7,000 days (>19 years) and 1,530 days (>4 years) of simulated data to develop personalised dosing policies. In an in vivo setting, allowing an RL algorithm to control a patient’s blood glucose for several years without any associated guarantee of safety would be unethical.

This work represents the first rigorous evaluation of offline RL for glucose control; contrasting the performance of a diverse range of state-of-the-art algorithms in a comprehensive cohort of virtual patients. Evaluation is per- formed for a combined 41,660 virtual days (>114 years) and the presented algorithms are evaluated inline with the current clinical guidance for assessing patient glucose control. This analysis holds a particular focus on exploring the practical and safety limitations of offline RL within hybrid closed loop

5

systems. In silico evaluation of this nature is essential in justifying trials of offline RL for glucose control in real patient populations.

3.1. Problem Formulation

The task of glucose control in hybrid closed loop systems can be modelled as a partially observable Markov decision process (POMDP) described by (S, A, Ω, T , R, O). In each timestep t, the agent in a single state s ∈S, describing the current environment, interacts with the environment via an action a ∈A and receives a reward r = R(s) ∈R specifying the optimality of the action, before transitioning into a new state s′ ∈S with transition probability P(s′|s, a) = T (s, a, s′).

In A Pomdp, The Agent Is Unable To

directly view the next state s′ ∈S, and instead receives an observation o ∈Ωdetermined by probability P(o′|s′, a) = O(o′, s′, a) which provides insight into the next state . In the glucose control setting, the state and action are defined by the patient blood glucose value gt and by the basal insulin dose it in that timestep. The partial observability of the state results from the inherent noise in the CGM devices used to make blood glucose measurements and the dependency of blood glucose values on historical data, such as ingested carbohydrates ct, bolus doses bt and previous blood glucose values . Contextual information was incorporated in this work by utilising

The Following State:

st = [gt, gt−1, gt−2, ..., gt−8, It, Mt].

(1)

This representation consists of a rolling window of measured blood glucose values updated every three minutes using the past four hours of blood glucose data spaced at 30-minute intervals, with gt being the blood glucose in the current timestep and and gt−8 being the blood glucose four hours prior. An estimation of the combined insulin activity of basal and bolus insulin (insulin- on-board) It and an estimation of carbohydrate activity Mt are also included

6

where N represents the number of prior timesteps the algorithm considers in its decision-making. This representation simplifies true insulin and carbohy- drate activity by assuming that they decay linearly to zero over a four hour period. This state was selected in place of the full sequence of blood glucose, carbohydrate and insulin data utilised in other approaches to reduce state dimensionality and to avoid modifying the offline RL methods to incorporate recurrency [15, 14, 39]. The reward for the agent was given by the negative of the Magni risk function, which models the clinical risk for a given blood glucose value . An additional penalty of -1e5 was added for blood glu- cose values beyond the physiologically feasible range of 10 to 1,000 mg/dl.

This modification is several orders of magnitude larger than the obtainable reward under the Magni risk function and was included as an incentive for the agent to not purposefully terminate the environment and avoid future negative reward. The parameters for the risk function are given as follows

Log (Gt)0.8353 −3.7932

2 .

(4)

A reward function of this form ensures that low blood glucose events are punished more severely than high blood glucose events; reflecting the greater immediate risk low blood glucose events pose to patient health. The reward is at a maximum when blood glucose is approximately in the centre of the target range (70-180 mg/dl).

This Reward Function Was Selected As Prior

work found it resulted in the greatest empirical performance , however alternative reward functions for glucose control are referenced in Tejedor et al. .

3.2. Offline Reinforcement Learning

RL algorithms learn the optimal series of actions to take in a given environmental state to maximise the agent’s total reward received.

This

state-action mapping is referred to as the agent’s policy and is updated from demonstrations of interactions with the chosen environment. Typically, en- vironmental demonstrations are generated in an online manner, in which the agent takes actions in the environment and updates its understanding in parallel. However in offline RL, the agent is incapable of environmental interaction during the training procedure and instead must rely on samples generated by a demonstrator in a retrospective or simulated dataset, such as a PID algorithm, to build an understanding of the POMDP .

Rl

methods can be broadly divided into model-free and model-based, which are

7

distinguished by the use of a dynamics model through which the transition probability is approximated. Alternatively, model-free approaches often esti- mate the Q-function, which defines the expected future reward of the agent when taking action a in state s and then continuing to make decisions using

(5)

where γ ∈[0, 1) weights the agent’s future reward. In the simplest form, an agent updates its approximation of the Q-function via the Bellman equation:

(6)

where α ∈(0, 1] is the learning rate. The agent then utilises the learned Q-function to update its policy, for example the online RL algorithm DQN selects the action in each state corresponding to the maximum expected

A

Q(s, a).

(7)

A central challenge of applying offline RL to real-world tasks is distributional shift, in which an offline RL agent encounters states significantly different from those observed in the training data . In these instances, extrapola- tion to out-of-distribution states can result in the erroneous overestimation of the Q-function and the selection of poor actions. This is particularly sig- nificant in safety-focused tasks such as glucose control. This work applies the following model-free offline RL approaches to reduce Q-function overes-

Timation:

• Batch Constrained Deep Q-learning (BCQ) modifies the DQN algorithm by constraining the agent to select similar actions and states to those observed in the training data .

In Addition To The Q-

function estimator, a variational auto-encoder is trained to generate similar actions to those in the training data when in a given state and diversity is incorporated by adjusting the generated actions using a perturbation model. State visitation is also constrained by applying a modified version of clipped double q-learning, whereby two separate

8

Q-function approximators are updated using the minimum of their es- timates . This adjustment reduces overestimation bias by minimis- ing the Q-function in each update. As a consequence of this reduction, high Q-values are preferentially assigned to states with low variance that have high visitation in the training dataset.

• Conservative Q-learning (CQL) expands on prior works, such as BCQ, by employing an alternative approach of addressing Q-function overestimation on out-of-distribution state-action tuples. CQL learns a conservative Q-function, which acts as a lower bound on the true Q-value for a given policy .

This Is Incorporated By Modifying

the traditional Q-function update to include an additional term which simultaneously minimises Q-value estimates on unseen state-action tu- ples, while maximising the Q-values of tuples observed in the dataset.

In theory, this change encourages the agent to preferentially select in- distribution actions by assigning them high Q-value estimates. • Twin Delayed DDPG with Behavioural Cloning (TD3-BC) modifies the established off-policy online RL method TD3, to include a behavioural cloning term in its loss; encouraging the policy to select actions observed in the training distribution . Beyond the afore- mentioned modification, TD3 utilises a number of methods to avoid Q-function overestimation such as clipped double Q-learning and Q- function smoothing.

3.3. Baselines

The performance of the offline RL methods were compared to two baseline algorithms: a tuned PID controller and the online RL method, recurrent soft actor-critic (SAC-RNN). The PID algorithm is a robust classical control mechanism capable of correcting for both short and long-term deviations from a target blood glucose value gtarget and operates in each timestep t

(8)

where kp, ki and kd are parameters to be set. To ensure the strongest com- parison, the parameters were personalised to each patient and were selected

9

using a grid-search method to maximise the reward collected over a 10-day test period. This algorithm is one of the most well used in hybrid closed loop systems , including the Medtronic 670g and 780g Guardian 3 sensors .

The SAC-RNN method represents one of the state-of-the-art algorithms in glucose control and was recently presented in Fox et al. . The method used in this paper is a variation of this implementation, using long-short- term memory (LSTM) layers in place of gated recurrent unit (GRU) layers.

This substitution was made as it was empirically found to improve algorith- mic performance and policy stability and thus provided a stronger baseline. Futher details of the implementation can be found in the provided repository.

3.4. Glucose Dynamics Simulation

The UVA/Padova T1D glucose dynamics model was used to generate data in this work, as it allowed control over the size and quality of the train- ing dataset . In addition, to providing a rigorous platform for evaluating the developed control algorithms without relying on unverified offline evalu- ation methods. This software simulates the human metabolic system using dietary models of glucose-insulin kinetics and is designed as a substitute for pre-clinical trials in the development and testing of T1D control algorithms . The simulator can model a cohort of 30 virtual patients (10 children, 10 adolescents and 10 adults) and their individualised responses to meal carbohydrates, basal/bolus insulin dosing and interaction with CGM/pump devices. For the purpose of this study, all virtual patients utilised a CGM with a three-minute sampling rate joined to a pump device. This sampling rate is the default for the simulator and is within the range of real-world sys- tems (1 to 15 minutes) . Few physiological processes and events relevant to T1D management occur over timeframes shorter than the CGM sampling rate, therefore control algorithms should perform comparably across the re- alistic range. Continuously valued basal insulin doses were selected by the RL/PID agent with bolusing administered using the following controller :

(9)

where gtarget = 144 mg/dl is the target blood glucose level and corresponds to the greatest reward in the Magni risk function, where the probability of high and low blood glucose events are at a minimum . In addition, CF and CR are patient-specific parameters and were chosen from Fox et al. .

10

Three meals and three snack events were included in the simulator with the time and quantity of carbohydrate ingestion for each event being modelled by a normal distribution.

3.5.1. Data Collection And Training

Each algorithm was trained on 1e5 samples of data collected over epochs of 10 days; the equivalent of 208 days of glucose data. Samples were collected from 30 simulated patients (10 children, 10 adolescents and 10 adults) and individually each patient’s data was used to train each algorithm across three seeds. Training and testing in adolescent and child populations is important for the adoption of T1D technology as these groups are typically more sus- ceptible to high blood glucose events and increased glucose variability .

The training data was collected using a PID algorithm tuned to achieve the maximum reward over a 10-day test period and noise was added to improve exploration within the generated dataset.

Basal Noise Was Introduced By

using an Ornstein-Uhlenbeck process . In addition, bolus noise was in- troduced by adding a 10% estimation error to the simulated carbohydrate intake; this more closely models the uncertainty observed in patient calcula- tions of bolus doses. The hyperparameters used for the offline RL algorithms were unchanged from their original implementations. This choice was made as hyperparameter optimisation would require patient interaction via the simulator to validate model performance; potentially harming the partici- pant in the process. Hyperparameter selection could also be performed using offline evaluation methods however this is outside the scope of this work.

The Percentage Time For Which Blood Glucose Mea-

surements fall within the healthy glucose range (70- 180 mg/dl).

With A Reduced Risk Of Developing Micro-Vascular

complications .

The Percentage Time For Which Blood Glucose Mea-

surements fall in the low blood glucose range (<70

Mg/Dl). Combined With Tir, This Can Additionally

act as an indirect measure of time spent in the high blood glucose range (>180 mg/dl).

The Relative Dispersion Of Blood Glucose Values

around their mean.

Increased Cv Is Linked To An

elevated risk of severe low blood glucose events (<54 mg/dl) and vascular tissue damage .

Failure

The percentage of test rollouts in which blood glu-

Cose Levels Reached Values <10 Mg/Dl Or >1000

mg/dl.

For Context, Blood Glucose <40 Mg/Dl Is

considered life-threatening and can result in major cardiovascular and cerebrovascular problems .

0%

Table 1: A description of the glycemic control metrics utilised for evaluation alongside their clinically recommended values. *For ages <25 this target decreases to >60%.

3.5.2. Evaluation

Performance was evaluated by monitoring blood glucose levels over a sim- ulated 10-day test period and aggregating the results over three test seeds per training seed to ensure sufficient variation in test scenarios. The metrics utilised for evaluation are given in Table 1 and were used in addition to the sum of reward for assessing algorithmic performance. Friedman rank tests and Wilcoxon signed-rank tests were used to assess significance between con- trol algorithm outcomes. In addition, the standard error between test seeds is presented alongside each measurement. Further tests were employed to identify the practical limitations of using offline RL algorithms in hybrid closed loop systems. This included evaluation on datasets with: 1) decreas- ing sample size, 2) demonstrations from suboptimal PID demonstrators and 3) sequences of missing CGM data caused by temporary sensor transmitter errors for extended periods. In addition, the potential of offline RL for safer blood glucose management was explored by engineering several common and

12

challenging control scenarios. These included patients with: 4) consistently overestimated bolus insulin for meals, 5) irregular meal schedules with greater uncertainty in meal times and 6) frequent erroneous low blood glucose read- ings caused by compression lows. Compression lows result from pressure on the CGM sensor insertion site and are caused by the redistribution of the interstitial fluid from which blood glucose is measured .

4. Results

4.1. Offline Reinforcement Learning vs. Baseline Control Methods A comparison of the described offline RL methods with baseline ap- proaches is detailed in Table 2. Of the methods presented, the offline RL algorithm TD3-BC achieved the best performance; obtaining the greatest reward and hence the lowest Magni risk over the evaluation period. In ad- dition, the algorithm yields a 3.7 ± 0.6% increase to TIR and a reduction in TBR when compared to the PID algorithm. The observed difference may in part be due to the inclusion of carbohydrate information in the state; pro- viding an early indication of when a sharp rise in blood glucose may occur.

The ability of RL algorithms to readily incorporate new sensor modalities without explicit programming is a significant advantage of the approach over non-machine learning based methods and could be utilised to incorporate a patient’s individualised response to exercise or stress if provided with the relevant sensor data.

This equates to almost an additional hour per day in which patients would experience an improved quality of life. BCQ also shows a similar level of improvement to TIR, however this is coupled with a 0.6±0.1% increase to TBR. The use of BCQ also resulted in an increase to CV of 1.6 ±0.4%. This value has been found clinically to fall within the region of 31.0% to 42.3% for people with T1D . An elevated CV should indicate an increased risk of low blood glucose events, which is evidenced with the BCQ algorithm.

The origin of the difference is most likely the use of the Magni risk function for defining reward, as this value does not consider blood glucose variability within its risk calculation.

0.00

Table 2: The mean performance of the offline RL algorithms: BCQ, CQL and TD3-BC against the online RL approach SAC-RNN and the control baseline PID. TD3-BC can be seen to significantly improve the proportion of TIR when compared to the PID and the SAC-RNN algorithms. This is done so without any associated increase in risk (reward) or TBR. Statistical significance was confirmed via a Friedman rank test for all glucose metrics (p < 0.05). †, ‡ and § indicate an offline RL, online RL and classical control algorithm respectively, with the best performing algorithm highlighted in bold.

The online RL algorithm SAC-RNN, performs comparably worse than the PID and offline RL approaches in almost all metrics; terminating the environment and thus harming the patient in 13.3% of the test rollouts. In this instance, CV does fall well within the recommended threshold of <36%, however this is likely a consequence of patients having high blood glucose for 61.0 ± 3.2% of the evaluation period and therefore being closer to their blood glucose equilibrium point . The performance of SAC-RNN in this work significantly differs from the results obtained previously under similar implementations [15, 63]. This difference is most likely a result of differ- ing evaluation methods. In this work, SAC-RNN was trained for the full duration and evaluated using the resulting weights, however in previous im- plementations performance was measured on a validation environment after each episode and the highest performing weights were selected for evalua- tion. One such online method reported that using the final weights in place of the best performing weights resulted in almost half of test rollouts ending in termination . This work elected to use the final weights for online evaluation, as it was concluded to be more indicative of how the algorithm would be utilised in a patient setting. Whereby, the algorithm would seek to continually adapt to changes in the patient’s lifestyle and blood glucose dynamics.

14

4.2. Offline Reinforcement Learning Performance by Patient Age (a) Adults (aged 26 to 68 years).

0.00

(b) Adolescents (aged 14 to 19 years).

0.00

(c) Children (aged 7 to 12 years).

0.00

Table 3: The mean glucose control performance for the individual patient models divided by age group, as specified in the UVA/Padova T1D Simulator. TD3-BC represents the only RL approach to perform better than the PID across the three patient groups. The greatest improvement is observed in adults where BCQ and TD3-BC yield a significant increase to time in the target range, equivalent to an improvement of 100 mins/day and 60 mins/day. Statistical significance was confirmed for all metrics via a Friedman rank test (p < 0.05). †, ‡ and § indicate an offline RL, online RL and classical control algorithm respectively, with the best performing algorithm highlighted in bold.

Table 3 presents a breakdown of glucose control performance when di- vided by patient age.

As In Table 2, The Offline Rl Algorithm Td3-Bc

performs the most consistently across the patient groups, achieving a greater reward to the PID across all categories. The greatest improvements to TIR are observed in adult patients, where BCQ and TD3-BC achieve an increase of 6.8 ± 0.7% and 4.2 ± 0.7% respectively. This observed difference is signif- icant enough to push TIR to within the recommended margin for that age

15

group, which if sustained would potentially lead to markedly better long-term health outcomes for that population. The results in the child group are also particularly promising, whereby the TD3-BC approach yields a 5.9 ± 0.4% increase to TIR and a 1.4 ± 0.1% reduction to TBR. Children represent one of the most challenging control groups within the T1D simulator and in real life, as evidenced by the significantly lower reward and greater glycemic variability obtained under the PID in that group. This control disparity is predominately due to differing insulin sensitivities between the age groups.

Insulin sensitivity has been identified to negatively correlate with a patient’s age and consequently smaller doses of insulin elicit greater blood glucose re- sponses in children and adolescents and require greater precision in insulin dosing . Achieving an improvement of this magnitude is encouraging for the transition of offline RL algorithms to real patient data in which blood glucose relationships are likely to be more complex and depend on a greater number of environmental factors. The TD3-BC algorithm was selected for further evaluation due to the high performance the approach achieved across the 30 virtual patients, in addition to its consistent safety profile.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.