Ultrasound Fetal Monitor Hardware

Ultrasound Images⋆

Kamil Mikolaj1,2, Manxi Lin1, Zahra Bashir2, Morten Bo Søndergaard Svendsen2, Martin Tolsgaard2, Anders Nymark1, and Aasa Feragen1

2 Cames Rigshospitalet

Abstract. Confounding information in the form of text or markings em- bedded in medical images can severely affect the training of diagnostic deep learning algorithms. However, data collected for clinical purposes often have such markings embedded in them. In dermatology, known ex- amples include drawings or rulers that are overrepresented in images of malignant lesions. In this paper, we encounter text and calipers placed on the images found in national databases containing fetal screening ul- trasound scans, which correlate with standard planes to be predicted. In order to utilize the vast amounts of data available in these databases, we develop and validate a series of methods for minimizing the confounding effects of embedded text and calipers on deep learning algorithms de- signed for ultrasound, using standard plane classification as a test case.

Figure: System Model & Simulation Flow for Ultrasound Fetal Monitor Hardware

Keywords: Removing confounding information · Model Bias · Fetal Ul- trasound · Standard Plane Classification.

Introduction

Clinical data is a great potential source of data for training medical imaging models: It can come in vast amounts, and represents the nature of data quality encountered in clinical practice. However, when data comes from the clinic, there is little control of the data generating process, and the data may be affected in ways that are suboptimal for training deep learning models. In particular, clinical images sometimes come with embedded text, markings, calipers or other annotations made by the clinician, and these likely carry information correlating with the predictive task at hand. It has recently been shown that markings, stickers and rulers present in dermatological images can confound predictors that aim to diagnose skin lesions . In this paper, we consider confounding information present in fetal ultrasound images from clinical screening. As shown in Fig. 1, these images often have text and calipers embedded in them, which can affect predictors trained on the images. As a case study, we use standard plane classification, which aims to automatically recognize those ultrasound planes required for particular types of screening tests during pregnancy.

Figure: System Model & Simulation Flow for Ultrasound Fetal Monitor Hardware

⋆Supported by organization x.

2

F. Author et al. Fig. 1. Examples of clinical ultrasound images with text and calipers, i.e. annotated coordinates for measuring anatomical objects, embedded into the image. Note that both the text labels and the caliper geometry carries information about the particular standard plane that the image contains.

Our contribution First, we quantify the confounding effect of text and calipers embedded in fetal ultrasound images used for training neural networks. Next, we quantitatively assess the success of different methods that aim to re- move these confounding effects, ranging from naïve to state of the art text in- painting methods developed for natural images. We show that even simple meth- ods that mask out the confounding information ensure improved generalization to images that do not contain confounding information.

Related Work

Recent work in dermatological imaging pointed out that pen markings or rulers that are often present in images of malignant skin lesions from clinical practice, were actually confounding skin lesion diagnosis performed by a CNN approved as a medical device . More precisely, it was shown that the predictive mod- els performed better on images with rulers and markings than on those without.

This discovery spurred research into methods for removing this confounding ef- fect, such as segmenting the lesion as a preprocessing step to avoid looking at context; inpainting stickers present on benign images to remove them from the training images ; inserting prior knowledge into the models ; or ad- versarially training the neural network to be unable to recognize whether the confounding information is present .

In our setting, the confounding information is typically embedded into the clinically relevant part of the image. As a result, we cannot apply methods that remove context by segmenting out the object of interest. Instead, we focus on validating both simple and more complex models for removing and inpainting confounding text and calipers.

Removing confounding information from fetal ultrasound images

Methodology

We assess the confounding effects of embedded text and calipers using standard plane classification for 3rd trimester growth scans as a test case. In order to assess the weight of the fetus, as is commonly done in the 3rd trimester, the clinician needs to obtain standard planes for the head, abdomen and femur. As a typical application is to recognize good standard planes from nonstandard planes that cannot be used for standardized measurements, we also include images that are not standard planes, which should ideally be classified as "Other".

We assess six different methods for removing confounding information. Ex- amples of images inpainted with the different methods are found in Fig. 2.

Simple Methods For Removing Confounding Effects

The initial four methods consist of first detecting the text and calipers and then replacing them in various ways. The text and calipers embedded in the clinically relevant part of the image is (in our training set) always yellow; these are detected via thresholding in hue, saturation, value (HSV) space to segment yellow features. The resulting mask is dilated with a 3x3 structuring element to enlarge the segmentation and connect neighboring elements.

Additionally, as can be seen in Fig. 1, most images contain some gray and blue text in the top right corner. To remove this, we first remove everything around the conical ultrasound field of view. A mask is obtained by thresholding the cone in HSV, finding the largest connected component and filling the holes, after which everything else can be masked out. For cases where some blue and grey text is on top of the field of view, the blue text is detected, and everything above is replaced by a black box.

We next consider various approaches to inpainting the yellow masks. Black box. In this first simple approach, the detected yellow mask is overlaid by black boxes spanned by the minimum and maximum x- and y- coordinates found within every connected component of the mask.

Replacing confounding information by noise. As the inpainted black boxes leaves clearly visible information on relative position, caliper geometry etc, we next replace the missing information with noise. For every connected compo- nent of the mask, we find its minimal bounding box as above, and expand the bounding box given by the coordinates by 5 pixels in each direction. Next, the contents of the box is replaced by noise as follows: The mean µ and standard deviation σ are computed from the values of those bounding box pixels which are not segmented as belonging to text/markings. Those pixels segmented as text/markings are then replaced with noise sampled from a normal distribution N(µ, σ/10). The scaling of the standard deviation was performed to reduce the visual effect of replacing image pixels with noise; the scaling factor selected from

1

100} by optimizing validation set accuracy.

4

F. Author et al. Fig. 2. Illustration of inpainting methods replacing both text and calipers on two example image patches.

Bilinear interpolation. Third, we apply bilinear interpolation to inpaint the mask. Fast marching inpainting. Next, we apply a fast marching inpainting method available in OpenCV to inpaint the mask.

2.2

Deep learning methods for removing confounding effects Next, we assess two different deep learning approaches that aim to generate confounder-free images. Due to the memory cost, the images are divided into 300 × 240 pixels patches for model training and inference.

U-Net. First, we train a U-net to generate images without confounding information. We regress the text-free image directly by designing the U-net so that its input and output both have three channels. The pixel intensity of the input images is rescaled to [−1, 1], and the network output is activated by the hyperbolic tangent function. The target labels are images inpainted by bilinear interpolation. We use the SGD optimizer with a learning rate of 0.001 and mon- itor the mean square root loss. The network is trained for 100 epochs with batch size 8.

GAN for generating text-free images. As an alternative, we train a GAN , which is a state-of-the-art model for scene text removal in natural images. We follow the training settings from the original work. The training set is synthe- sized by placing the markings detected by thresholding randomly on the images inpainted by bilinear interpolation. This is a common way to construct datasets in scene text removal.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.